merge: transport hardening (register rows 51-53) — BundleLockedException curated to Commons/MapFault, fleet-wide MS SQL unlock-throttle window, import-session gauge + cap Warning
This commit is contained in:
@@ -7,7 +7,7 @@ The Configuration Database component is the exclusive EF Core data-access layer
|
||||
The component lives in `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/` and is central-only — site clusters never load it. Its responsibilities break down into four areas:
|
||||
|
||||
- **DbContext + Fluent mappings** — `ScadaBridgeDbContext` maps ~30 Commons POCO entity types to SQL Server using `IEntityTypeConfiguration<T>` classes in `Configurations/`, registered wholesale via `modelBuilder.ApplyConfigurationsFromAssembly(...)`.
|
||||
- **Repository implementations** — eleven scoped repositories implement the interfaces declared in Commons, covering every domain area from template authoring to audit log ingest.
|
||||
- **Repository implementations** — scoped repositories implement the interfaces declared in Commons, covering every domain area from template authoring to audit log ingest.
|
||||
- **Config-change audit** — `AuditService` implements `IAuditService`, staging an `AuditLogEntry` into the change tracker so it commits atomically with the entity change; `AuditCorrelationContext` threads a `BundleImportId` through `AsyncLocal<T>` so bundle-import audit rows are correlated without cross-contaminating concurrent import sessions.
|
||||
- **Partition maintenance** — `AuditLogPartitionMaintenance` implements `IPartitionMaintenance`, rolling `pf_AuditLog_Month` forward by issuing `ALTER PARTITION FUNCTION … SPLIT RANGE` for each missing future monthly boundary.
|
||||
|
||||
@@ -91,9 +91,11 @@ The predecessor `UX_AuditLog_EventId` — a single-column unique index on `[PRIM
|
||||
|
||||
**`SiteCallEntityTypeConfiguration`** maps `SiteCall` to `dbo.SiteCalls` with a `TrackedOperationId` PK stored as `varchar(36)` (GUID in `"D"` format) so the column shape matches the wire format and the site SQLite store — one consistent format for operational debugging.
|
||||
|
||||
**`BundleUnlockAttemptEntityTypeConfiguration`** maps `BundleUnlockAttempt` to the table of the same name — the fleet-wide sliding window behind Transport's per-client bundle-unlock throttle (register row 52; the counters used to be per-node and per-process, so the effective cap was N × the configured limit). Operational history, not audit: non-partitioned, `[PRIMARY]` filegroup, no DB-role restriction. One named index, `IX_BundleUnlockAttempt_Client_Attempted`, serves both the windowed count and the application's opportunistic prune, which is bounded and best-effort — a store failure in this path is deliberately fail-open at the Transport layer, so nothing here may block an unlock. The table deliberately records no bundle identity; the durable record of a failed unlock is the `BundleImportUnlockFailed` audit row.
|
||||
|
||||
### Repository implementations
|
||||
|
||||
All eleven repositories follow the same shape: they take `ScadaBridgeDbContext` by constructor injection, work with Commons POCO types, and never commit — callers invoke `SaveChangesAsync()` to commit the unit of work.
|
||||
All repositories follow the same shape: they take `ScadaBridgeDbContext` by constructor injection, work with Commons POCO types, and never commit — callers invoke `SaveChangesAsync()` to commit the unit of work.
|
||||
|
||||
**`AuditLogRepository`** is the most specialized. Its `InsertIfNotExistsAsync` bypasses the change tracker and issues raw interpolated SQL because the computed columns must not appear in the INSERT column list:
|
||||
|
||||
@@ -152,7 +154,7 @@ services.AddConfigurationDatabase(
|
||||
configuration["ScadaBridge:Database:ConfigurationDb"]!);
|
||||
```
|
||||
|
||||
This registers `ScadaBridgeDbContext` as scoped (with the runtime encryption overload), all eleven repository interfaces bound to their implementations, `IAuditCorrelationContext` → `AuditCorrelationContext`, `IAuditService` → `AuditService`, `IInstanceLocator` → `InstanceLocator`, `IPartitionMaintenance` → `AuditLogPartitionMaintenance`, and the Data Protection key ring persisted to the database.
|
||||
This registers `ScadaBridgeDbContext` as scoped (with the runtime encryption overload), every repository interface bound to its implementation, `IAuditCorrelationContext` → `AuditCorrelationContext`, `IAuditService` → `AuditService`, `IInstanceLocator` → `InstanceLocator`, `IPartitionMaintenance` → `AuditLogPartitionMaintenance`, and the Data Protection key ring persisted to the database.
|
||||
|
||||
The obsolete zero-argument overload throws `InvalidOperationException` at startup (marked `error: true` on the `[Obsolete]` attribute) so a misconfigured host fails fast with a clear message rather than silently producing an empty DI registration.
|
||||
|
||||
|
||||
@@ -49,6 +49,11 @@ After `LoadAsync` validates and decrypts a bundle, the plaintext content is stor
|
||||
|
||||
Without this, an abandoned wizard or a plain CLI preview held one of the eight `MaxConcurrentImportSessions` slots — and the decrypted bundle content — for the full TTL; eight of them locked out every import on the active node (deferred register row 43, observed live).
|
||||
|
||||
**Session-cap observability (register row 53).** That leak was only ever inferable from rotating test failures, because nothing counted open sessions and nothing logged a rejection. Two signals now close the gap:
|
||||
|
||||
- **Gauge `scadabridge.central.transport.import_sessions.open`** — open (unexpired) sessions on this node, on the `ZB.MOM.WW.ScadaBridge` meter, which is already in `SiteServiceRegistration.ObservedMeters` (the `ZbTelemetryOptions.Meters` allowlist; an unlisted meter exports nothing, silently). The provider is registered by `BundleSessionEvictionService.StartAsync` and identity-checked-cleared on `StopAsync`, mirroring the StoreAndForward queue-depth gauge. `IBundleSessionStore.OpenSessionCount` deliberately excludes expired-but-unswept entries — counting the raw dictionary would report the node as pinned at the cap for up to a sweep interval after a burst had actually drained, which is the exact false positive this instrument exists to rule out. Transport is registered only in the host's Central branch, so the gauge reports 0 on a site node.
|
||||
- **Warning log on rejection** — `BundleSessionStore.Open` logs the open count, the cap, and the TTL when it rejects. It fires only on a real cap hit: a store full of *expired* sessions is reclaimed first and logs nothing. A count that stays pinned at the cap across repeated warnings is a leak; an occasional warning is genuine concurrency, and the fix there is to raise `ScadaBridge:Transport:MaxConcurrentImportSessions` (each slot pins up to `MaxBundleSizeMb` of decrypted plaintext, so the cap is a memory budget).
|
||||
|
||||
### Conflict resolution and `BundleImportId` correlation
|
||||
|
||||
`PreviewAsync` compares each bundle artifact against the target environment using `ArtifactDiff`, classifying items as `Identical`, `Modified`, `New`, or `Blocker`. The operator assigns a `ResolutionAction` (`Add`, `Overwrite`, `Skip`, `Rename`) per item. `ApplyAsync` honours those resolutions in a single EF transaction and threads a new `BundleImportId` GUID through every per-entity audit row via the scoped `IAuditCorrelationContext`. This makes every configuration row written by a bundle import queryable as a group from the audit log.
|
||||
@@ -122,10 +127,18 @@ Passphrase lockout operates at two levels: per-bundle (3-strike counter in `Bund
|
||||
|
||||
The two levels interact in a specific order, and it is load-bearing (register row 44 — the limiter had been registered, injected and unit-tested but never called, so only the per-bundle lockout was real):
|
||||
|
||||
1. **Per-bundle lockout first.** An already-locked bundle throws `BundleLockedException` without registering a rate-limiter attempt, so hammering a locked bundle cannot spend the client's hourly budget.
|
||||
1. **Per-bundle lockout first.** An already-locked bundle throws `BundleLockedException` (Commons — register row 51 moved it out of `Transport.Import` for the same reason the throttle exception lives there: `ManagementService` does not reference `Transport`, so a lockout used to reach the CLI as `MapFault`'s generic "An internal error occurred" plus a correlation id) without registering a rate-limiter attempt, so hammering a locked bundle cannot spend the client's hourly budget.
|
||||
2. **Per-IP throttle next, before the decrypt.** Over budget throws `BundleUnlockThrottledException` (Commons, so the management surface can catch it and return a curated message). Because it precedes the decrypt, a denial burns **no** per-bundle strike — otherwise the throttle would be a way to lock out a bundle someone is legitimately importing. Every unlock attempt is registered, successful ones included: the documented budget counts attempts, not failures.
|
||||
3. Only then the AES-GCM decrypt, whose failure increments the per-bundle counter as before.
|
||||
|
||||
**The per-IP window is fleet-wide (register row 52).** It used to be a process-local `ConcurrentDictionary` on each central node, which had two consequences worth naming: the effective fleet cap was N × `MaxUnlockAttemptsPerIpPerHour` per client (a client that failed over to the other node got a fresh budget), and a node restart cleared the window outright. The window now lives in the central MS SQL `BundleUnlockAttempt` table — the HA store both central nodes already share, so this needs no new infrastructure, no new port and no consensus protocol. `BundleUnlockRateLimiter` keeps the policy; `SqlBundleUnlockAttemptWindow` is the store, resolving a scoped `IBundleUnlockAttemptRepository` from its own scope per call (it is a singleton, like the limiter). Three properties are load-bearing:
|
||||
|
||||
- **It fails OPEN.** A store error logs a Warning naming the client and the cap, and *permits* the attempt. This is a best-effort rate cap layered over a surface that is already authenticated and Administrator-gated, so the availability of legitimate imports outranks the cap — a database blip must not make the import wizard unusable. Falling back to the old process-local window instead was considered and rejected: it would still block a legitimate operator's 11th import during an outage, which is precisely the outcome fail-open exists to prevent. The Warning is the operator's signal that the fleet cap is not currently being enforced.
|
||||
- **Accuracy is deliberately approximate.** The count-then-insert pair is not atomic across the two nodes, so genuinely simultaneous attempts can overshoot the cap by up to the number in flight. A conditional single-statement insert would close that at the cost of provider-specific raw SQL; the overshoot is bounded and self-corrects within the window.
|
||||
- **Rows are pruned opportunistically, not by a background service.** Transport owns no central scheduler, so the prune rides the unlock path: claimed by CAS at most once per 15 minutes per process, bounded to 1000 rows per pass, retaining `2 ×` the window so a prune racing a count can never delete a row the count still needs, and wrapped in a try/catch that logs and swallows — retention maintenance must never fail the operator's actual unlock. This is the same shape `DeploymentService` uses for terminal deployment records.
|
||||
|
||||
Behaviour the operator sees is otherwise unchanged: same config key, same semantics ("attempts, not failures"), same ordering relative to the per-bundle lockout, same `<unattributed>` bucket.
|
||||
|
||||
The client key is the caller's remote IP: the Central UI wizard reads it from `CircuitClientAddressProvider` (a scoped service that snapshots `Connection.RemoteIpAddress` at circuit establishment, since `HttpContext` is gone by the time a circuit submits a passphrase — the same pattern as `CookieAuthenticationStateProvider`), and the management endpoint stamps it on `ManagementEnvelope.ClientIp` for the CLI `transport preview` / `transport import` paths. A caller with no attributable address is counted against a single **shared** bucket rather than skipping the throttle — "unknown client" degrades to a stricter budget, never to none. As with the login throttle (`Component-Security.md`), per-client isolation is only real when ForwardedHeaders is configured for the fronting proxy; behind an untrusted proxy every caller collapses into the proxy's bucket.
|
||||
|
||||
```csharp
|
||||
@@ -255,11 +268,11 @@ A template's scripts reference a `SharedScript` or `ExternalSystem` that exists
|
||||
|
||||
### Passphrase lockout
|
||||
|
||||
After 3 wrong passphrase attempts against the same bundle (keyed by `ContentHash`), `BundleImporter.LoadAsync` throws `BundleLockedException`. The session is unusable. Re-upload the bundle file to get a new session with a fresh counter. A `BundleImportUnlockFailed` audit row is written on each failed attempt.
|
||||
After 3 wrong passphrase attempts against the same bundle (keyed by `ContentHash`), `BundleImporter.LoadAsync` throws `BundleLockedException`. The session is unusable. Re-upload the bundle file to get a new session with a fresh counter. A `BundleImportUnlockFailed` audit row is written on each failed attempt. Both the Central UI and the CLI surface the lockout reason verbatim — the exception lives in `Commons/Types/Transport/` so `ManagementActor` can curate it into a `ManagementCommandException` (register row 51).
|
||||
|
||||
### Unlock throttled ("Too many bundle unlock attempts")
|
||||
|
||||
`BundleUnlockThrottledException` — this **client** (not this bundle) has used its `MaxUnlockAttemptsPerIpPerHour` budget within the trailing hour. Re-uploading does not help; the budget is keyed by IP and rolls forward on a sliding window, and the counters are process-local (a central node restart clears them, deliberately, so an outage doesn't strand legitimate operators). The denial burns no per-bundle strike, so the bundle itself is still importable once the window frees up. Raise `ScadaBridge:Transport:MaxUnlockAttemptsPerIpPerHour` if a legitimate workflow (many encrypted bundles in one sitting) trips it routinely.
|
||||
`BundleUnlockThrottledException` — this **client** (not this bundle) has used its `MaxUnlockAttemptsPerIpPerHour` budget within the trailing hour. Re-uploading does not help; the budget is keyed by IP, rolls forward on a sliding window, and is **fleet-wide and restart-durable** (register row 52 — see below), so failing over to the other central node does not help either. The denial burns no per-bundle strike, so the bundle itself is still importable once the window frees up. Raise `ScadaBridge:Transport:MaxUnlockAttemptsPerIpPerHour` if a legitimate workflow (many encrypted bundles in one sitting) trips it routinely.
|
||||
|
||||
### BundleImportAlarmScriptUnresolved / BundleImportCompositionUnresolved warnings
|
||||
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
-- Register row 52: the fleet-wide bundle-unlock throttle window.
|
||||
--
|
||||
-- Creates the central [BundleUnlockAttempt] table that both central nodes share, so
|
||||
-- ScadaBridge:Transport:MaxUnlockAttemptsPerIpPerHour is a fleet cap rather than a
|
||||
-- per-node, per-process one. Operational history, not audit: non-partitioned, standard
|
||||
-- [PRIMARY] filegroup, no DB-role restriction, and rows are pruned opportunistically by
|
||||
-- the application once they age past twice the trailing window.
|
||||
--
|
||||
-- Run with QUOTED_IDENTIFIER ON. Several tables in this database carry filtered indexes
|
||||
-- and persisted computed columns, and SQL Server refuses DDL/DML against them with error
|
||||
-- 1934 when the session has QUOTED_IDENTIFIER OFF -- which is exactly what `sqlcmd` gives
|
||||
-- you by default (pass -I, or keep this header). The SETs sit in their own batch so they
|
||||
-- are in force when the statements after the GO parse.
|
||||
SET QUOTED_IDENTIFIER ON;
|
||||
SET ANSI_NULLS ON;
|
||||
GO
|
||||
|
||||
BEGIN TRANSACTION;
|
||||
IF NOT EXISTS (
|
||||
SELECT * FROM [__EFMigrationsHistory]
|
||||
WHERE [MigrationId] = N'20260815154803_AddBundleUnlockAttemptTable'
|
||||
)
|
||||
BEGIN
|
||||
EXEC(N'CREATE TABLE [BundleUnlockAttempt] (
|
||||
[Id] bigint NOT NULL IDENTITY,
|
||||
[ClientKey] varchar(64) NOT NULL,
|
||||
[AttemptedAtUtc] datetime2 NOT NULL,
|
||||
CONSTRAINT [PK_BundleUnlockAttempt] PRIMARY KEY ([Id])
|
||||
)');
|
||||
|
||||
EXEC(N'CREATE INDEX [IX_BundleUnlockAttempt_Client_Attempted]
|
||||
ON [BundleUnlockAttempt] ([ClientKey], [AttemptedAtUtc])');
|
||||
END;
|
||||
|
||||
IF NOT EXISTS (
|
||||
SELECT * FROM [__EFMigrationsHistory]
|
||||
WHERE [MigrationId] = N'20260815154803_AddBundleUnlockAttemptTable'
|
||||
)
|
||||
BEGIN
|
||||
INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion])
|
||||
VALUES (N'20260815154803_AddBundleUnlockAttemptTable', N'10.0.7');
|
||||
END;
|
||||
|
||||
COMMIT;
|
||||
GO
|
||||
@@ -123,6 +123,9 @@ The configuration database stores all central system data, organized by domain a
|
||||
- **Deployment Records**: Deployment history per instance (timestamp, user, status, deployed configuration snapshot).
|
||||
- **System-Wide Artifact Deployment Records**: Deployment history for shared artifacts (timestamp, user, artifact type, status).
|
||||
|
||||
### Transport
|
||||
- **Bundle Unlock Attempts**: One row per permitted bundle-unlock passphrase attempt (`ClientKey`, `AttemptedAtUtc`), forming the fleet-wide sliding window behind Transport's `MaxUnlockAttemptsPerIpPerHour` throttle — see `Component-Transport.md` (register row 52). Operational history, not audit: non-partitioned, no DB-role restriction, and rows are pruned opportunistically by the application once they age past twice the window. **Denied** attempts are deliberately not recorded (a denial burns no budget), and no bundle identity is stored — the durable record of a failed unlock is the `BundleImportUnlockFailed` audit row. Central-only: written only by the Transport component, which the host registers for central roles.
|
||||
|
||||
### Audit Logging
|
||||
- **Audit Log Entries**: Append-only audit trail (timestamp, user, action, entity type, entity ID, entity name, state as JSON). Stores only the after-state — change history is reconstructed by comparing consecutive entries. Entries are never modified or deleted. No retention policy — retained indefinitely. Indexed on timestamp, user, entity type, entity ID, and action for efficient filtering.
|
||||
|
||||
|
||||
@@ -134,7 +134,7 @@ flowchart TD
|
||||
class T muted
|
||||
```
|
||||
|
||||
The component is central-hosted. It is registered in `ZB.MOM.WW.ScadaBridge.Host` for central roles only, never for site roles — even when a bundle carries site-scoped config (T18), Transport only writes the central configuration database, never the site clusters. All persistence flows through existing audited repository interfaces in `ZB.MOM.WW.ScadaBridge.ConfigurationDatabase` — the component does not call `DbContext.SaveChangesAsync` directly. `BundleSessionStore` is in-process on the active central node (matching Blazor Server circuit affinity): 30-minute TTL, eviction on expiry, 3-strike passphrase lockout per session, and a `MaxConcurrentImportSessions` cap (default 8; arch-review 05 Task 24 — startup-validated as `> 0` by `TransportOptionsValidator`, since a zero/negative cap would reject every import session forever) bounding the decrypted-content footprint since each open session pins a fully-decrypted bundle in memory.
|
||||
The component is central-hosted. It is registered in `ZB.MOM.WW.ScadaBridge.Host` for central roles only, never for site roles — even when a bundle carries site-scoped config (T18), Transport only writes the central configuration database, never the site clusters. All persistence flows through existing audited repository interfaces in `ZB.MOM.WW.ScadaBridge.ConfigurationDatabase` — the component does not call `DbContext.SaveChangesAsync` directly. `BundleSessionStore` is in-process on the active central node (matching Blazor Server circuit affinity): 30-minute TTL, eviction on expiry, 3-strike passphrase lockout per session, and a `MaxConcurrentImportSessions` cap (default 8; arch-review 05 Task 24 — startup-validated as `> 0` by `TransportOptionsValidator`, since a zero/negative cap would reject every import session forever) bounding the decrypted-content footprint since each open session pins a fully-decrypted bundle in memory. **The cap is observable (register row 53):** open (unexpired) sessions are published as the `scadabridge.central.transport.import_sessions.open` gauge on the `ZB.MOM.WW.ScadaBridge` meter — already in the `ObservedMeters` allowlist, so no telemetry config change is needed — and a rejection at the cap logs at Warning with the current count, the cap and the TTL. Reclaiming a store full of *expired* sessions is not a cap hit and logs nothing.
|
||||
|
||||
**Session release is explicit, not TTL-only (register row 43).** `ApplyAsync` releases the session it applied (zero-then-remove, T-007), and `IBundleImporter.Abandon(sessionId)` does the same for a session that will never be applied. Every abandonment path calls it: the Central UI wizard on new upload / Back-to-Upload / `ResetSessionState` / lockout reset / `Dispose` (navigation away or circuit teardown), the CLI `transport preview` handler (a preview never applies), and the CLI `transport import` handler for the aborts that precede apply (blocker rows, unresolved mappings). `Abandon` is idempotent — double release, an unknown id, and an already-expired id are all silent no-ops — and since session ids are random GUIDs handed only to their opener, abandoning the id you hold can never evict another operator's in-progress import. Relying on the TTL alone was an availability *and* secret-lifetime defect: eight abandoned wizards exhausted `MaxConcurrentImportSessions` and locked out every import on the active node for 30 minutes, each still holding its decrypted bundle content.
|
||||
|
||||
@@ -336,7 +336,9 @@ Imports are all-or-nothing per bundle. A bundle either applies fully or not at a
|
||||
|
||||
- **AES-256-GCM** for content encryption with **PBKDF2-SHA256 / 600 000 iterations** (OWASP 2023+ guidance), per-bundle random salt, random IV per encryption. The GCM auth tag is verified before decryption — a wrong passphrase fails cleanly.
|
||||
- **Passphrase never persisted.** It lives only inside the export/import service call path and is discarded after use. No environment variable, no log line.
|
||||
- **Failed-unlock rate limit:** per-bundle 3-strike lockout (keyed by `ContentHash`, server-side); per-IP-per-hour cap (`MaxUnlockAttemptsPerIpPerHour`, default 10, configurable) to deter brute force against a stolen bundle. Each failed attempt produces a `BundleImportUnlockFailed` audit row. The two are evaluated in order — bundle lockout, then per-IP throttle, then decrypt — so hammering an already-locked bundle spends no hourly budget, and a throttle denial (`BundleUnlockThrottledException`, fail-closed with a curated message on both the UI and CLI surfaces) burns **no** per-bundle strike, since it never reaches the decryptor. Successful unlocks count against the hourly budget too: it caps attempts, not failures. The client key is the caller's remote IP — from `CircuitClientAddressProvider` (snapshotted at Blazor circuit establishment, since `HttpContext` is unavailable mid-circuit) for the Central UI, and from `ManagementEnvelope.ClientIp` for the CLI; a caller with no attributable address shares a single stricter bucket rather than bypassing the throttle. Per-client isolation is only real behind a ForwardedHeaders-trusted proxy, the same caveat the login throttle carries in `Component-Security.md`. (Register row 44: the limiter existed, was configured and unit-tested, but was never invoked until this was wired.)
|
||||
- **Failed-unlock rate limit:** per-bundle 3-strike lockout (keyed by `ContentHash`, server-side); per-IP-per-hour cap (`MaxUnlockAttemptsPerIpPerHour`, default 10, configurable) to deter brute force against a stolen bundle. Each failed attempt produces a `BundleImportUnlockFailed` audit row. The two are evaluated in order — bundle lockout, then per-IP throttle, then decrypt — so hammering an already-locked bundle spends no hourly budget, and a throttle denial (`BundleUnlockThrottledException`, fail-closed with a curated message on both the UI and CLI surfaces) burns **no** per-bundle strike, since it never reaches the decryptor. Both typed exceptions live in `Commons/Types/Transport/` — `ManagementService` references Commons but not Transport, so a type declared beside the importer cannot be curated by `MapFault` and reaches the CLI as a generic internal-error message (register rows 44 and 51). Successful unlocks count against the hourly budget too: it caps attempts, not failures. The client key is the caller's remote IP — from `CircuitClientAddressProvider` (snapshotted at Blazor circuit establishment, since `HttpContext` is unavailable mid-circuit) for the Central UI, and from `ManagementEnvelope.ClientIp` for the CLI; a caller with no attributable address shares a single stricter bucket rather than bypassing the throttle. Per-client isolation is only real behind a ForwardedHeaders-trusted proxy, the same caveat the login throttle carries in `Component-Security.md`. (Register row 44: the limiter existed, was configured and unit-tested, but was never invoked until this was wired.)
|
||||
|
||||
**The per-IP window is fleet-wide and restart-durable (register row 52).** It is persisted in the central MS SQL `BundleUnlockAttempt` table — the HA store both central nodes already share — rather than in each node's process memory, so the documented cap is a fleet cap and not N × the cap, and a node restart no longer clears it. `BundleUnlockRateLimiter` still owns the policy; `SqlBundleUnlockAttemptWindow` is the store behind the `IBundleUnlockAttemptWindow` seam, and `IBundleUnlockAttemptRepository` (Commons interface, ConfigurationDatabase implementation) is the persistence. Three deliberate trades: **the throttle fails OPEN** — a store error logs a Warning and permits the attempt, because this is a best-effort rate cap over an already-authenticated Administrator-gated surface and the availability of legitimate imports outranks the cap; **accuracy is approximate** — count-then-insert is not atomic across nodes, so simultaneous attempts can overshoot by the number in flight, which is bounded and self-corrects within the window; and **rows are pruned opportunistically on the unlock path** (CAS-claimed at most every 15 minutes per process, bounded to 1000 rows, retaining twice the window, best-effort) rather than by a new background service, since Transport owns no central scheduler.
|
||||
- **Bundle size cap** on upload (default 100 MB, configurable) to bound memory.
|
||||
- **In-transit:** existing HTTPS to the Central UI; no new channel.
|
||||
- **Audit trail is the chain of custody.** Every export, every import (including aborted ones at validation), and every unlock failure is audit-logged with source env, content hash, encrypted yes/no, and artifact summary.
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
namespace ZB.MOM.WW.ScadaBridge.Commons.Entities.Transport;
|
||||
|
||||
/// <summary>
|
||||
/// One recorded bundle-unlock passphrase attempt, written to the central
|
||||
/// <c>BundleUnlockAttempt</c> table. Rows form the trailing-hour sliding window the
|
||||
/// Transport component's per-client unlock throttle
|
||||
/// (<c>TransportOptions.MaxUnlockAttemptsPerIpPerHour</c>) is evaluated against.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// Register row 52: the throttle's counters used to be per-node and per-process, so the
|
||||
/// effective fleet cap was N x limit per client and a node restart cleared the window.
|
||||
/// Central MS SQL is the store both central nodes already share, so putting the window
|
||||
/// there makes the cap fleet-wide and restart-durable without new infrastructure.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Operational history, NOT audit — the table is non-partitioned, lives on the standard
|
||||
/// <c>[PRIMARY]</c> filegroup and carries no DB-role restriction. Rows are pruned
|
||||
/// opportunistically once they age past the window; they are not a durable record of
|
||||
/// who tried to open which bundle (that is the <c>BundleImportUnlockFailed</c> audit row's
|
||||
/// job, and note this table deliberately records no bundle identity at all).
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Persistence-ignorant POCO; the EF Core mapping lives in the Configuration Database
|
||||
/// component. <see cref="AttemptedAtUtc"/> is UTC, like every timestamp in the system.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class BundleUnlockAttempt
|
||||
{
|
||||
/// <summary>Surrogate identity key assigned by the store.</summary>
|
||||
public long Id { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// The throttle bucket the attempt was counted against — the caller's remote IP, or
|
||||
/// the shared <c><unattributed></c> bucket when no address could be attributed.
|
||||
/// Sized for an IPv6 literal with a scope id.
|
||||
/// </summary>
|
||||
public required string ClientKey { get; set; }
|
||||
|
||||
/// <summary>The UTC instant at which the unlock attempt was registered.</summary>
|
||||
public DateTime AttemptedAtUtc { get; set; }
|
||||
}
|
||||
+46
@@ -0,0 +1,46 @@
|
||||
namespace ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
|
||||
|
||||
/// <summary>
|
||||
/// Register row 52: read/write access to the central <c>BundleUnlockAttempt</c> table,
|
||||
/// the fleet-wide sliding window behind the Transport component's per-client bundle-unlock
|
||||
/// throttle. Both central nodes share this store, so the documented
|
||||
/// <c>MaxUnlockAttemptsPerIpPerHour</c> cap is a fleet cap rather than a per-process one.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The throttle is a best-effort rate cap layered over an already-authenticated,
|
||||
/// role-gated surface. Callers are expected to treat a store failure as fail-open (the
|
||||
/// availability of legitimate imports outranks the cap), so no method here is required to
|
||||
/// be transactional with respect to the caller's own work.
|
||||
/// </remarks>
|
||||
public interface IBundleUnlockAttemptRepository
|
||||
{
|
||||
/// <summary>
|
||||
/// Counts the attempts recorded for <paramref name="clientKey"/> at or after
|
||||
/// <paramref name="sinceUtc"/> — the trailing-window depth the cap is compared against.
|
||||
/// </summary>
|
||||
/// <param name="clientKey">Throttle bucket (client IP, or the shared unattributed bucket).</param>
|
||||
/// <param name="sinceUtc">Inclusive UTC lower bound of the trailing window.</param>
|
||||
/// <param name="ct">Cancellation token.</param>
|
||||
/// <returns>The number of attempts inside the window.</returns>
|
||||
Task<int> CountAttemptsSinceAsync(string clientKey, DateTime sinceUtc, CancellationToken ct = default);
|
||||
|
||||
/// <summary>
|
||||
/// Appends one attempt row for <paramref name="clientKey"/>.
|
||||
/// </summary>
|
||||
/// <param name="clientKey">Throttle bucket (client IP, or the shared unattributed bucket).</param>
|
||||
/// <param name="attemptedAtUtc">UTC instant of the attempt.</param>
|
||||
/// <param name="ct">Cancellation token.</param>
|
||||
/// <returns>A task that completes when the row is committed.</returns>
|
||||
Task RecordAttemptAsync(string clientKey, DateTime attemptedAtUtc, CancellationToken ct = default);
|
||||
|
||||
/// <summary>
|
||||
/// Deletes attempt rows older than <paramref name="beforeUtc"/>, up to
|
||||
/// <paramref name="maxRows"/> in one bounded pass. Bounded so an opportunistic prune on
|
||||
/// a request path can never turn into an unbounded delete.
|
||||
/// </summary>
|
||||
/// <param name="beforeUtc">Exclusive UTC upper bound — rows strictly older than this are eligible.</param>
|
||||
/// <param name="maxRows">Maximum rows to delete in this pass; must be positive.</param>
|
||||
/// <param name="ct">Cancellation token.</param>
|
||||
/// <returns>The number of rows deleted.</returns>
|
||||
Task<int> PruneAttemptsOlderThanAsync(DateTime beforeUtc, int maxRows, CancellationToken ct = default);
|
||||
}
|
||||
@@ -18,6 +18,15 @@ public interface IBundleSessionStore
|
||||
/// <summary>Removes all sessions whose expiry has passed.</summary>
|
||||
void EvictExpired();
|
||||
|
||||
/// <summary>
|
||||
/// Register row 53: the number of sessions currently held that have not yet expired.
|
||||
/// Backs the <c>scadabridge.central.transport.import_sessions.open</c> gauge, so it
|
||||
/// must be cheap and side-effect-free (it is read from the metrics collection thread)
|
||||
/// and must NOT count expired-but-unswept entries — otherwise the gauge would read
|
||||
/// as "at the cap" for up to a sweep interval after a burst drained.
|
||||
/// </summary>
|
||||
int OpenSessionCount { get; }
|
||||
|
||||
/// <summary>
|
||||
/// T-003: returns the current unlock-failure count for a bundle keyed by its
|
||||
/// content hash. The counter is server-owned so a second tab / CLI caller
|
||||
|
||||
@@ -118,6 +118,14 @@ public static class ScadaBridgeTelemetry
|
||||
/// <summary>Provider that yields the live StoreAndForward queue depth; set by a later task.</summary>
|
||||
private static Func<long>? _queueDepthProvider;
|
||||
|
||||
/// <summary>
|
||||
/// Register row 53: provider that yields the number of currently open (unexpired)
|
||||
/// Transport bundle-import sessions, registered by the Transport component's
|
||||
/// session-eviction hosted service. Null on a site node (Transport is central-only),
|
||||
/// where the gauge simply reports 0.
|
||||
/// </summary>
|
||||
private static Func<long>? _importSessionCountProvider;
|
||||
|
||||
#pragma warning disable IDE0052 // Held to keep the observable gauges alive for the meter's lifetime.
|
||||
/// <summary>Gauge reporting the number of currently open site connections.</summary>
|
||||
private static readonly ObservableGauge<long> _siteConnectionUp =
|
||||
@@ -138,6 +146,20 @@ public static class ScadaBridgeTelemetry
|
||||
() => Interlocked.Read(ref _liveAlarmAggregatorsActive),
|
||||
unit: "1",
|
||||
description: "Number of per-site live-alarm aggregators currently running on the active central node.");
|
||||
|
||||
/// <summary>
|
||||
/// Register row 53: gauge reporting the number of open Transport bundle-import
|
||||
/// sessions on this central node. Each open session pins a fully decrypted bundle
|
||||
/// in memory and consumes one of <c>TransportOptions.MaxConcurrentImportSessions</c>
|
||||
/// (default 8), so a series that plateaus at the cap is the signature of leaked
|
||||
/// sessions — the failure mode register row 43 could only be inferred from rotating
|
||||
/// test failures because nothing counted them.
|
||||
/// </summary>
|
||||
private static readonly ObservableGauge<long> _transportImportSessionsOpen =
|
||||
Meter.CreateObservableGauge<long>("scadabridge.central.transport.import_sessions.open",
|
||||
() => Volatile.Read(ref _importSessionCountProvider) is { } p ? p() : 0L,
|
||||
unit: "1",
|
||||
description: "Number of currently open bundle-import sessions on this central node.");
|
||||
#pragma warning restore IDE0052
|
||||
|
||||
// ---------------- Emit helpers ----------------
|
||||
@@ -242,4 +264,42 @@ public static class ScadaBridgeTelemetry
|
||||
// delegate, so a stale stop cannot clobber a successor's provider.
|
||||
Interlocked.CompareExchange(ref _queueDepthProvider, null, provider);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Register row 53: registers the provider the open-import-sessions gauge reads on
|
||||
/// each observation. Called by the Transport component's session-eviction hosted
|
||||
/// service on start. A null provider is ignored so the gauge falls back to 0.
|
||||
/// The callback is invoked synchronously on the metrics collection thread, so it
|
||||
/// must stay cheap — the Transport implementation reads a
|
||||
/// <see cref="System.Collections.Concurrent.ConcurrentDictionary{TKey,TValue}"/> it
|
||||
/// already owns.
|
||||
/// </summary>
|
||||
/// <param name="provider">A callback returning the current open-session count.</param>
|
||||
public static void SetImportSessionCountProvider(Func<long> provider)
|
||||
{
|
||||
if (provider is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
Volatile.Write(ref _importSessionCountProvider, provider);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Clears the open-import-sessions provider, but only if the currently registered
|
||||
/// provider is the exact <paramref name="provider"/> delegate passed in
|
||||
/// (reference-equal compare-and-clear), so a late stop of a superseded instance
|
||||
/// cannot stomp a newer registrant. Mirrors <see cref="ClearQueueDepthProvider"/>.
|
||||
/// </summary>
|
||||
/// <param name="provider">The provider delegate to remove; ignored unless it is the
|
||||
/// one currently registered.</param>
|
||||
public static void ClearImportSessionCountProvider(Func<long> provider)
|
||||
{
|
||||
if (provider is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
Interlocked.CompareExchange(ref _importSessionCountProvider, null, provider);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,44 @@
|
||||
namespace ZB.MOM.WW.ScadaBridge.Commons.Types.Transport;
|
||||
|
||||
/// <summary>
|
||||
/// T-003: thrown by <c>IBundleImporter.LoadAsync</c> when an encrypted bundle has
|
||||
/// exceeded the configured failed-unlock attempt limit
|
||||
/// (<c>TransportOptions.MaxUnlockAttemptsPerSession</c>, default 3). The lockout is tracked
|
||||
/// server-side keyed by <see cref="BundleManifest.ContentHash"/>, so a second tab / CLI caller
|
||||
/// re-uploading the same bytes hits the same counter and cannot side-step the limit.
|
||||
/// <para>
|
||||
/// Distinct from <see cref="BundleUnlockThrottledException"/> on purpose: this one means
|
||||
/// "this BUNDLE has burned its strikes" and is keyed by the bundle's content hash; that one
|
||||
/// means "this CLIENT has attempted too many unlocks, across any bundles" and is keyed by
|
||||
/// the caller's IP.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Register row 51: lives in Commons rather than beside the importer for the same reason
|
||||
/// <see cref="BundleUnlockThrottledException"/> does — it is part of the
|
||||
/// <c>IBundleImporter.LoadAsync</c> contract, which is declared here, and the management
|
||||
/// surface (which references Commons but not Transport) must be able to catch it to return
|
||||
/// a clear, curated error instead of a generic internal-failure message plus a correlation id.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public sealed class BundleLockedException : Exception
|
||||
{
|
||||
/// <summary>Number of recorded unlock failures for this bundle.</summary>
|
||||
public int FailedAttempts { get; }
|
||||
|
||||
/// <summary>SHA-256 (hex) of the bundle's content bytes, the lockout's tracking key.</summary>
|
||||
public string BundleContentHash { get; }
|
||||
|
||||
/// <summary>
|
||||
/// Initializes a new <see cref="BundleLockedException"/>.
|
||||
/// </summary>
|
||||
/// <param name="bundleContentHash">SHA-256 hex from <see cref="BundleManifest.ContentHash"/>.</param>
|
||||
/// <param name="failedAttempts">Number of failures recorded against this bundle.</param>
|
||||
public BundleLockedException(string bundleContentHash, int failedAttempts)
|
||||
: base(
|
||||
$"Bundle is locked after {failedAttempts} failed unlock attempts. "
|
||||
+ "Wait for the lockout window to expire or re-export the bundle to obtain a new content hash.")
|
||||
{
|
||||
BundleContentHash = bundleContentHash ?? throw new ArgumentNullException(nameof(bundleContentHash));
|
||||
FailedAttempts = failedAttempts;
|
||||
}
|
||||
}
|
||||
+46
@@ -0,0 +1,46 @@
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
using Microsoft.EntityFrameworkCore.Metadata.Builders;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Transport;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Configurations;
|
||||
|
||||
/// <summary>
|
||||
/// Maps the <see cref="BundleUnlockAttempt"/> POCO to the central
|
||||
/// <c>BundleUnlockAttempt</c> table — the fleet-wide sliding window behind the Transport
|
||||
/// component's per-client bundle-unlock throttle (register row 52). Operational history,
|
||||
/// NOT audit, so the table is non-partitioned, standard <c>[PRIMARY]</c> filegroup, no
|
||||
/// DB-role restriction.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// One named index, <c>IX_BundleUnlockAttempt_Client_Attempted</c>, serves both access
|
||||
/// paths: the window count seeks on <c>ClientKey</c> and ranges on
|
||||
/// <c>AttemptedAtUtc</c>, and the opportunistic prune scans the same index by time. A
|
||||
/// second time-only index was deliberately not added — the table is kept small by the
|
||||
/// prune, so the leading-column seek is cheap enough for the prune to share.
|
||||
/// </remarks>
|
||||
public class BundleUnlockAttemptEntityTypeConfiguration : IEntityTypeConfiguration<BundleUnlockAttempt>
|
||||
{
|
||||
/// <summary>
|
||||
/// Configures the EF Core entity type mapping for <see cref="BundleUnlockAttempt"/>.
|
||||
/// </summary>
|
||||
/// <param name="builder">The entity type builder to configure.</param>
|
||||
public void Configure(EntityTypeBuilder<BundleUnlockAttempt> builder)
|
||||
{
|
||||
builder.ToTable("BundleUnlockAttempt");
|
||||
|
||||
// Surrogate long identity assigned by the store.
|
||||
builder.HasKey(a => a.Id);
|
||||
|
||||
// Bucket key: an IP literal or the "<unattributed>" sentinel. 64 chars covers an
|
||||
// IPv6 address with a scope id; ASCII, so non-unicode (varchar) like every other
|
||||
// catalog-bounded column in the schema.
|
||||
builder.Property(a => a.ClientKey).HasMaxLength(64).IsUnicode(false).IsRequired();
|
||||
|
||||
builder.Property(a => a.AttemptedAtUtc).IsRequired();
|
||||
|
||||
// Window index — seeks the bucket, ranges over the trailing hour, and is
|
||||
// reused by the prune's time scan. Name locked for migration discoverability.
|
||||
builder.HasIndex(a => new { a.ClientKey, a.AttemptedAtUtc })
|
||||
.HasDatabaseName("IX_BundleUnlockAttempt_Client_Attempted");
|
||||
}
|
||||
}
|
||||
+2121
File diff suppressed because it is too large
Load Diff
+41
@@ -0,0 +1,41 @@
|
||||
using System;
|
||||
using Microsoft.EntityFrameworkCore.Migrations;
|
||||
|
||||
#nullable disable
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Migrations
|
||||
{
|
||||
/// <inheritdoc />
|
||||
public partial class AddBundleUnlockAttemptTable : Migration
|
||||
{
|
||||
/// <inheritdoc />
|
||||
protected override void Up(MigrationBuilder migrationBuilder)
|
||||
{
|
||||
migrationBuilder.CreateTable(
|
||||
name: "BundleUnlockAttempt",
|
||||
columns: table => new
|
||||
{
|
||||
Id = table.Column<long>(type: "bigint", nullable: false)
|
||||
.Annotation("SqlServer:Identity", "1, 1"),
|
||||
ClientKey = table.Column<string>(type: "varchar(64)", unicode: false, maxLength: 64, nullable: false),
|
||||
AttemptedAtUtc = table.Column<DateTime>(type: "datetime2", nullable: false)
|
||||
},
|
||||
constraints: table =>
|
||||
{
|
||||
table.PrimaryKey("PK_BundleUnlockAttempt", x => x.Id);
|
||||
});
|
||||
|
||||
migrationBuilder.CreateIndex(
|
||||
name: "IX_BundleUnlockAttempt_Client_Attempted",
|
||||
table: "BundleUnlockAttempt",
|
||||
columns: new[] { "ClientKey", "AttemptedAtUtc" });
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void Down(MigrationBuilder migrationBuilder)
|
||||
{
|
||||
migrationBuilder.DropTable(
|
||||
name: "BundleUnlockAttempt");
|
||||
}
|
||||
}
|
||||
}
|
||||
+25
@@ -1716,6 +1716,31 @@ namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Migrations
|
||||
b.ToTable("TemplateScripts");
|
||||
});
|
||||
|
||||
modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.Commons.Entities.Transport.BundleUnlockAttempt", b =>
|
||||
{
|
||||
b.Property<long>("Id")
|
||||
.ValueGeneratedOnAdd()
|
||||
.HasColumnType("bigint");
|
||||
|
||||
SqlServerPropertyBuilderExtensions.UseIdentityColumn(b.Property<long>("Id"));
|
||||
|
||||
b.Property<DateTime>("AttemptedAtUtc")
|
||||
.HasColumnType("datetime2");
|
||||
|
||||
b.Property<string>("ClientKey")
|
||||
.IsRequired()
|
||||
.HasMaxLength(64)
|
||||
.IsUnicode(false)
|
||||
.HasColumnType("varchar(64)");
|
||||
|
||||
b.HasKey("Id");
|
||||
|
||||
b.HasIndex("ClientKey", "AttemptedAtUtc")
|
||||
.HasDatabaseName("IX_BundleUnlockAttempt_Client_Attempted");
|
||||
|
||||
b.ToTable("BundleUnlockAttempt", (string)null);
|
||||
});
|
||||
|
||||
modelBuilder.Entity("ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Entities.AuditLogRow", b =>
|
||||
{
|
||||
b.Property<Guid>("EventId")
|
||||
|
||||
+70
@@ -0,0 +1,70 @@
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Transport;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Repositories;
|
||||
|
||||
/// <summary>
|
||||
/// EF Core implementation of <see cref="IBundleUnlockAttemptRepository"/> over the central
|
||||
/// <c>BundleUnlockAttempt</c> table (register row 52). Plain reads/writes against the
|
||||
/// shared <see cref="ScadaBridgeDbContext"/>; the prune uses <c>ExecuteDeleteAsync</c> so
|
||||
/// no rows are materialised into the change tracker.
|
||||
/// </summary>
|
||||
public class BundleUnlockAttemptRepository : IBundleUnlockAttemptRepository
|
||||
{
|
||||
private readonly ScadaBridgeDbContext _context;
|
||||
|
||||
/// <summary>
|
||||
/// Initializes a new instance of the <see cref="BundleUnlockAttemptRepository"/> class.
|
||||
/// </summary>
|
||||
/// <param name="context">The EF Core database context.</param>
|
||||
public BundleUnlockAttemptRepository(ScadaBridgeDbContext context)
|
||||
{
|
||||
_context = context ?? throw new ArgumentNullException(nameof(context));
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task<int> CountAttemptsSinceAsync(
|
||||
string clientKey, DateTime sinceUtc, CancellationToken ct = default)
|
||||
{
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(clientKey);
|
||||
|
||||
return await _context.BundleUnlockAttempts
|
||||
.AsNoTracking()
|
||||
.Where(a => a.ClientKey == clientKey && a.AttemptedAtUtc >= sinceUtc)
|
||||
.CountAsync(ct);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task RecordAttemptAsync(
|
||||
string clientKey, DateTime attemptedAtUtc, CancellationToken ct = default)
|
||||
{
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(clientKey);
|
||||
|
||||
_context.BundleUnlockAttempts.Add(new BundleUnlockAttempt
|
||||
{
|
||||
ClientKey = clientKey,
|
||||
AttemptedAtUtc = attemptedAtUtc,
|
||||
});
|
||||
await _context.SaveChangesAsync(ct);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task<int> PruneAttemptsOlderThanAsync(
|
||||
DateTime beforeUtc, int maxRows, CancellationToken ct = default)
|
||||
{
|
||||
if (maxRows < 1)
|
||||
{
|
||||
throw new ArgumentOutOfRangeException(nameof(maxRows), "Prune batch size must be at least 1.");
|
||||
}
|
||||
|
||||
// Bounded by construction: OrderBy + Take caps the delete regardless of how much
|
||||
// history accumulated while the prune was failing or the node was down, so an
|
||||
// opportunistic sweep on a request path can never become an unbounded delete.
|
||||
return await _context.BundleUnlockAttempts
|
||||
.Where(a => a.AttemptedAtUtc < beforeUtc)
|
||||
.OrderBy(a => a.AttemptedAtUtc)
|
||||
.Take(maxRows)
|
||||
.ExecuteDeleteAsync(ct);
|
||||
}
|
||||
}
|
||||
@@ -13,6 +13,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Schemas;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Scripts;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Entities.SecuredWrites;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Transport;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Security;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates;
|
||||
@@ -152,6 +153,10 @@ public class ScadaBridgeDbContext : DbContext, IDataProtectionKeyContext
|
||||
/// <summary>Gets the set of hourly KPI rollups (pre-aggregated long-range trend backbone).</summary>
|
||||
public DbSet<KpiRollupHourly> KpiRollupHourly => Set<KpiRollupHourly>();
|
||||
|
||||
// Transport (bundle export/import)
|
||||
/// <summary>Gets the set of recorded bundle-unlock attempts — the fleet-wide sliding window behind the per-client unlock throttle (register row 52).</summary>
|
||||
public DbSet<BundleUnlockAttempt> BundleUnlockAttempts => Set<BundleUnlockAttempt>();
|
||||
|
||||
// Data Protection Keys (for shared ASP.NET Data Protection across nodes)
|
||||
/// <summary>Gets the set of data protection keys.</summary>
|
||||
public DbSet<DataProtectionKey> DataProtectionKeys => Set<DataProtectionKey>();
|
||||
|
||||
@@ -87,6 +87,10 @@ public static class ServiceCollectionExtensions
|
||||
services.AddScoped<ISecuredWriteRepository, SecuredWriteRepository>();
|
||||
services.AddScoped<ISharedSchemaRepository, SharedSchemaRepository>();
|
||||
services.AddScoped<IKpiHistoryRepository, KpiHistoryRepository>();
|
||||
// Register row 52: the fleet-wide bundle-unlock throttle window. Central-only in
|
||||
// practice (Transport is registered in the host's Central branch), but registered
|
||||
// here with every other repository so the ConfigurationDatabase surface stays uniform.
|
||||
services.AddScoped<IBundleUnlockAttemptRepository, BundleUnlockAttemptRepository>();
|
||||
// Auth re-arch: inbound API keys are no longer persisted in SQL Server —
|
||||
// the repository now exposes only API-method access, so a plain scoped
|
||||
// registration suffices (no peppered-hasher accessor to wire).
|
||||
|
||||
@@ -3567,6 +3567,17 @@ public class ManagementActor : ReceiveActor
|
||||
{
|
||||
throw new ManagementCommandException(ex.Message);
|
||||
}
|
||||
catch (BundleLockedException ex)
|
||||
{
|
||||
// Register row 51: the per-bundle 3-strike lockout. LoadAsync raises
|
||||
// it from inside its own CryptographicException handler, so without
|
||||
// this arm the operator saw MapFault's generic internal-error message
|
||||
// plus a correlation id, never the reason. Listed ahead of the
|
||||
// CryptographicException arm to keep the read order matching the
|
||||
// importer's throw order (the two types are unrelated, so the
|
||||
// ordering is documentation, not dispatch).
|
||||
throw new ManagementCommandException(ex.Message);
|
||||
}
|
||||
catch (System.Security.Cryptography.CryptographicException)
|
||||
{
|
||||
throw new ManagementCommandException("Wrong passphrase or bundle was tampered.");
|
||||
@@ -3632,6 +3643,17 @@ public class ManagementActor : ReceiveActor
|
||||
{
|
||||
throw new ManagementCommandException(ex.Message);
|
||||
}
|
||||
catch (BundleLockedException ex)
|
||||
{
|
||||
// Register row 51: the per-bundle 3-strike lockout. LoadAsync raises
|
||||
// it from inside its own CryptographicException handler, so without
|
||||
// this arm the operator saw MapFault's generic internal-error message
|
||||
// plus a correlation id, never the reason. Listed ahead of the
|
||||
// CryptographicException arm to keep the read order matching the
|
||||
// importer's throw order (the two types are unrelated, so the
|
||||
// ordering is documentation, not dispatch).
|
||||
throw new ManagementCommandException(ex.Message);
|
||||
}
|
||||
catch (System.Security.Cryptography.CryptographicException)
|
||||
{
|
||||
throw new ManagementCommandException("Wrong passphrase or bundle was tampered.");
|
||||
|
||||
@@ -293,11 +293,18 @@ public sealed class BundleImporter : IBundleImporter
|
||||
// operator is legitimately importing.
|
||||
// Registered on every unlock attempt (successful ones included) because the
|
||||
// documented budget counts attempts, not failures.
|
||||
//
|
||||
// Register row 52: the async overload consults the FLEET-WIDE window (central
|
||||
// MS SQL) so the cap is not multiplied by the number of central nodes and does
|
||||
// not reset on a node restart. It fails open on a store error — see
|
||||
// BundleUnlockRateLimiter's remarks.
|
||||
var throttleKey = string.IsNullOrWhiteSpace(clientKey)
|
||||
? UnattributedClientKey
|
||||
: clientKey;
|
||||
var perIpCap = _options.Value.MaxUnlockAttemptsPerIpPerHour;
|
||||
if (!_unlockRateLimiter.TryRegisterAttempt(throttleKey, perIpCap))
|
||||
if (!await _unlockRateLimiter
|
||||
.TryRegisterAttemptAsync(throttleKey, perIpCap, ct)
|
||||
.ConfigureAwait(false))
|
||||
{
|
||||
_logger?.LogWarning(
|
||||
"Bundle unlock throttled: client {ClientKey} exceeded {Cap} attempts in the trailing hour.",
|
||||
|
||||
@@ -1,31 +0,0 @@
|
||||
namespace ZB.MOM.WW.ScadaBridge.Transport.Import;
|
||||
|
||||
/// <summary>
|
||||
/// T-003: thrown by <see cref="BundleImporter.LoadAsync"/> when an encrypted bundle has
|
||||
/// exceeded the configured failed-unlock attempt limit
|
||||
/// (<see cref="TransportOptions.MaxUnlockAttemptsPerSession"/>). The lockout is tracked
|
||||
/// server-side keyed by <c>BundleManifest.ContentHash</c>, so a second tab / CLI caller
|
||||
/// re-uploading the same bytes hits the same counter and cannot side-step the limit.
|
||||
/// </summary>
|
||||
public sealed class BundleLockedException : Exception
|
||||
{
|
||||
/// <summary>Number of recorded unlock failures for this bundle.</summary>
|
||||
public int FailedAttempts { get; }
|
||||
|
||||
/// <summary>SHA-256 (hex) of the bundle's content bytes, the lockout's tracking key.</summary>
|
||||
public string BundleContentHash { get; }
|
||||
|
||||
/// <summary>
|
||||
/// Initializes a new <see cref="BundleLockedException"/>.
|
||||
/// </summary>
|
||||
/// <param name="bundleContentHash">SHA-256 hex from <c>BundleManifest.ContentHash</c>.</param>
|
||||
/// <param name="failedAttempts">Number of failures recorded against this bundle.</param>
|
||||
public BundleLockedException(string bundleContentHash, int failedAttempts)
|
||||
: base(
|
||||
$"Bundle is locked after {failedAttempts} failed unlock attempts. "
|
||||
+ "Wait for the lockout window to expire or re-export the bundle to obtain a new content hash.")
|
||||
{
|
||||
BundleContentHash = bundleContentHash ?? throw new ArgumentNullException(nameof(bundleContentHash));
|
||||
FailedAttempts = failedAttempts;
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,7 @@
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Transport;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Observability;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Transport.Import;
|
||||
|
||||
@@ -21,6 +22,15 @@ internal sealed class BundleSessionEvictionService : BackgroundService
|
||||
private readonly IBundleSessionStore _sessionStore;
|
||||
private readonly ILogger<BundleSessionEvictionService> _logger;
|
||||
|
||||
/// <summary>
|
||||
/// Register row 53: the open-session gauge callback, held in a field so
|
||||
/// <see cref="StopAsync"/> can pass the SAME delegate reference to
|
||||
/// <see cref="ScadaBridgeTelemetry.ClearImportSessionCountProvider"/> — the clear is
|
||||
/// an identity-checked compare-and-clear, so a stale stop cannot unregister a newer
|
||||
/// instance's provider. Mirrors StoreAndForwardService's queue-depth registration.
|
||||
/// </summary>
|
||||
private readonly Func<long> _openSessionCountProvider;
|
||||
|
||||
/// <summary>
|
||||
/// Initializes a new instance of <see cref="BundleSessionEvictionService"/>.
|
||||
/// </summary>
|
||||
@@ -32,6 +42,35 @@ internal sealed class BundleSessionEvictionService : BackgroundService
|
||||
{
|
||||
_sessionStore = sessionStore ?? throw new ArgumentNullException(nameof(sessionStore));
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
_openSessionCountProvider = () => _sessionStore.OpenSessionCount;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Register row 53: registers the open-import-sessions gauge provider before the
|
||||
/// sweep loop starts. This service is the Transport component's only lifecycle seam
|
||||
/// (the session store itself is a plain singleton), and it is registered by
|
||||
/// <c>AddTransport</c>, which only the Central branch of the host calls — so the
|
||||
/// gauge reports 0 on a site node, where there are no import sessions to count.
|
||||
/// </summary>
|
||||
/// <param name="cancellationToken">Host startup cancellation token.</param>
|
||||
/// <returns>A task that completes when the base <see cref="BackgroundService"/> has started.</returns>
|
||||
public override Task StartAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
ScadaBridgeTelemetry.SetImportSessionCountProvider(_openSessionCountProvider);
|
||||
return base.StartAsync(cancellationToken);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Deregisters this instance's gauge provider on graceful stop so the gauge falls
|
||||
/// back to 0 rather than reading through a dead store. The clear is identity-checked,
|
||||
/// so a late stop cannot clobber a successor that already re-registered.
|
||||
/// </summary>
|
||||
/// <param name="cancellationToken">Host shutdown cancellation token.</param>
|
||||
/// <returns>A task that completes when the base <see cref="BackgroundService"/> has stopped.</returns>
|
||||
public override async Task StopAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
ScadaBridgeTelemetry.ClearImportSessionCountProvider(_openSessionCountProvider);
|
||||
await base.StopAsync(cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
|
||||
@@ -1,4 +1,6 @@
|
||||
using System.Collections.Concurrent;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Transport;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Transport;
|
||||
@@ -48,6 +50,7 @@ public sealed class BundleSessionStore : IBundleSessionStore
|
||||
|
||||
private readonly TimeProvider _timeProvider;
|
||||
private readonly IOptions<TransportOptions> _options;
|
||||
private readonly ILogger<BundleSessionStore> _logger;
|
||||
|
||||
/// <summary>T-003: per-bundle unlock-failure entry with expiry.</summary>
|
||||
private sealed class UnlockFailureRecord
|
||||
@@ -61,12 +64,42 @@ public sealed class BundleSessionStore : IBundleSessionStore
|
||||
/// </summary>
|
||||
/// <param name="options">Transport options. <see cref="TransportOptions.BundleSessionTtlMinutes"/> is also used as the TTL for the T-003 per-bundle unlock-failure tracker.</param>
|
||||
/// <param name="timeProvider">Time provider used to evaluate session expiry.</param>
|
||||
public BundleSessionStore(IOptions<TransportOptions> options, TimeProvider timeProvider)
|
||||
/// <param name="logger">Optional logger. Register row 53: a rejection at <see cref="TransportOptions.MaxConcurrentImportSessions"/> is logged at Warning, so a session leak is diagnosable from the node log instead of being inferred from rotating test failures.</param>
|
||||
public BundleSessionStore(
|
||||
IOptions<TransportOptions> options,
|
||||
TimeProvider timeProvider,
|
||||
ILogger<BundleSessionStore>? logger = null)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
_options = options;
|
||||
_ = options.Value;
|
||||
_timeProvider = timeProvider ?? throw new ArgumentNullException(nameof(timeProvider));
|
||||
_logger = logger ?? NullLogger<BundleSessionStore>.Instance;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public int OpenSessionCount
|
||||
{
|
||||
get
|
||||
{
|
||||
// Register row 53: count only unexpired entries. _sessions.Count would
|
||||
// include expired-but-unswept sessions (the eviction sweep runs once a
|
||||
// minute), so the gauge would read "pinned at the cap" for up to a sweep
|
||||
// interval after a burst had actually drained — exactly the false signal
|
||||
// this instrument exists to rule out. Read-only: the gauge callback runs
|
||||
// on the metrics collection thread and must not mutate the store.
|
||||
var now = _timeProvider.GetUtcNow();
|
||||
var count = 0;
|
||||
foreach (var kv in _sessions)
|
||||
{
|
||||
if (kv.Value.ExpiresAt > now)
|
||||
{
|
||||
count++;
|
||||
}
|
||||
}
|
||||
|
||||
return count;
|
||||
}
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
@@ -82,6 +115,17 @@ public sealed class BundleSessionStore : IBundleSessionStore
|
||||
EvictExpired();
|
||||
if (_sessions.Count >= cap)
|
||||
{
|
||||
// Register row 53: this rejection used to be silent apart from the
|
||||
// exception the caller saw. A cap hit is either genuine concurrency
|
||||
// (many operators importing at once) or a session leak like register
|
||||
// row 43's — the count-vs-cap pair plus the recurrence rate in the
|
||||
// log is what distinguishes them.
|
||||
_logger.LogWarning(
|
||||
"Bundle import session rejected: {OpenSessions} of {Cap} concurrent import sessions are open "
|
||||
+ "(ScadaBridge:Transport:MaxConcurrentImportSessions). Sessions release on apply, cancel or "
|
||||
+ "the {TtlMinutes}-minute TTL; a count that stays pinned at the cap indicates leaked sessions.",
|
||||
_sessions.Count, cap, _options.Value.BundleSessionTtlMinutes);
|
||||
|
||||
throw new InvalidOperationException(
|
||||
$"Too many concurrent import sessions ({cap}). Complete or cancel an in-progress import before starting another.");
|
||||
}
|
||||
|
||||
@@ -1,35 +0,0 @@
|
||||
namespace ZB.MOM.WW.ScadaBridge.Transport.Import;
|
||||
|
||||
/// <summary>
|
||||
/// Thrown by <see cref="BundleImporter.LoadAsync"/> when the caller
|
||||
/// has exceeded the configured per-IP-per-hour unlock attempt cap
|
||||
/// (<see cref="TransportOptions.MaxUnlockAttemptsPerIpPerHour"/>). The 429-equivalent
|
||||
/// signal: the caller must wait for the trailing-hour window to roll forward before
|
||||
/// another passphrase attempt is accepted.
|
||||
/// </summary>
|
||||
public sealed class BundleUnlockRateLimitedException : Exception
|
||||
{
|
||||
/// <summary>
|
||||
/// Rate-limit key the limiter rejected the attempt against — the caller IP when
|
||||
/// supplied, or the bundle's content hash as the architectural fallback (the
|
||||
/// importer has no <c>IHttpContext</c> dependency by design).
|
||||
/// </summary>
|
||||
public string ClientKey { get; }
|
||||
|
||||
/// <summary>Per-window cap that was reached.</summary>
|
||||
public int Limit { get; }
|
||||
|
||||
/// <summary>
|
||||
/// Initializes a new <see cref="BundleUnlockRateLimitedException"/>.
|
||||
/// </summary>
|
||||
/// <param name="clientKey">The rate-limit key that exceeded its budget.</param>
|
||||
/// <param name="limit">The configured per-window cap.</param>
|
||||
public BundleUnlockRateLimitedException(string clientKey, int limit)
|
||||
: base(
|
||||
$"Bundle unlock rate limit reached ({limit} attempts per hour). "
|
||||
+ "Wait for the trailing-hour window to expire before retrying.")
|
||||
{
|
||||
ClientKey = clientKey ?? throw new ArgumentNullException(nameof(clientKey));
|
||||
Limit = limit;
|
||||
}
|
||||
}
|
||||
@@ -1,26 +1,37 @@
|
||||
using System.Collections.Concurrent;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Transport.Import;
|
||||
|
||||
/// <summary>
|
||||
/// In-memory sliding-window rate limiter for bundle-unlock passphrase
|
||||
/// attempts, keyed by client IP. The design doc (§11) declares a per-IP-per-hour cap
|
||||
/// (default 10) as a brute-force defence against a stolen bundle; this class is the
|
||||
/// minimal server-side implementation.
|
||||
/// Sliding-window rate limiter for bundle-unlock passphrase attempts, keyed by client IP.
|
||||
/// The design doc (§11) declares a per-IP-per-hour cap (default 10) as a brute-force
|
||||
/// defence against a stolen bundle; this class owns that policy.
|
||||
/// <para>
|
||||
/// Algorithm: each key (an IP string, or any opaque caller identifier) holds a queue
|
||||
/// of attempt timestamps. <see cref="TryRegisterAttempt"/> first prunes entries older
|
||||
/// than the configured window, then either appends the current timestamp and returns
|
||||
/// <c>true</c> if the count is still under the threshold, or refuses to append and
|
||||
/// returns <c>false</c> if appending would cross it. The trailing-hour count is the
|
||||
/// queue length post-prune.
|
||||
/// <b>The window is fleet-wide when an <see cref="IBundleUnlockAttemptWindow"/> is
|
||||
/// supplied (register row 52).</b> The counters used to be per-node AND per-process, so
|
||||
/// the effective cap was N x limit per client across the central pair and a node restart
|
||||
/// silently cleared the window. <see cref="TryRegisterAttemptAsync"/> now consults the
|
||||
/// shared central-MS-SQL window instead, which both central nodes already share. The
|
||||
/// production composition root always supplies one.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Storage is a process-local <see cref="ConcurrentDictionary{TKey,TValue}"/>. The
|
||||
/// counters do not survive a host restart — that is by design: a restart resets the
|
||||
/// brute-force window in favour of legitimate operators after an outage. Persisting
|
||||
/// the counters would require a multi-node consensus story the simple in-memory
|
||||
/// design avoids.
|
||||
/// <b>Fail-open is deliberate.</b> A store failure in the throttle path logs a Warning and
|
||||
/// PERMITS the attempt. This is a best-effort rate cap layered over a surface that is
|
||||
/// already authenticated and role-gated (Administrator-only bundle import), so the
|
||||
/// availability of legitimate imports outranks the cap: a database blip must not make the
|
||||
/// import wizard unusable. Falling back to the process-local window instead was considered
|
||||
/// and rejected — it would still block a legitimate operator's 11th import during an
|
||||
/// outage, which is the outcome fail-open exists to prevent.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Algorithm (the process-local fallback used when no shared window is registered): each
|
||||
/// key holds a queue of attempt timestamps. <see cref="TryRegisterAttempt"/> first prunes
|
||||
/// entries older than the configured window, then either appends the current timestamp and
|
||||
/// returns <c>true</c> if the count is still under the threshold, or refuses to append and
|
||||
/// returns <c>false</c> if appending would cross it. Either way a REJECTED attempt is never
|
||||
/// appended, so a denial burns no budget — the same contract the shared window implements.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Thread-safety: the per-key queue is protected by a per-key lock taken inside the
|
||||
@@ -39,10 +50,13 @@ public sealed class BundleUnlockRateLimiter
|
||||
private readonly ConcurrentDictionary<string, AttemptBucket> _buckets = new(StringComparer.OrdinalIgnoreCase);
|
||||
private readonly TimeProvider _timeProvider;
|
||||
private readonly TimeSpan _window;
|
||||
private readonly IBundleUnlockAttemptWindow? _sharedWindow;
|
||||
private readonly ILogger<BundleUnlockRateLimiter> _logger;
|
||||
|
||||
/// <summary>
|
||||
/// Initializes a new <see cref="BundleUnlockRateLimiter"/> using the documented
|
||||
/// 1-hour trailing window and the system clock. Suitable for production DI.
|
||||
/// 1-hour trailing window and the system clock, with no shared window (process-local
|
||||
/// counters only).
|
||||
/// </summary>
|
||||
public BundleUnlockRateLimiter() : this(TimeProvider.System, DefaultWindow)
|
||||
{
|
||||
@@ -50,11 +64,28 @@ public sealed class BundleUnlockRateLimiter
|
||||
|
||||
/// <summary>
|
||||
/// Initializes a new <see cref="BundleUnlockRateLimiter"/> with an injected clock
|
||||
/// (for deterministic tests) and a custom trailing window.
|
||||
/// (for deterministic tests) and a custom trailing window, with no shared window.
|
||||
/// </summary>
|
||||
/// <param name="timeProvider">Clock used for both timestamping new attempts and pruning expired ones.</param>
|
||||
/// <param name="window">Trailing window over which attempts are counted (typically 1 hour).</param>
|
||||
public BundleUnlockRateLimiter(TimeProvider timeProvider, TimeSpan window)
|
||||
: this(timeProvider, window, sharedWindow: null, logger: null)
|
||||
{
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Initializes a new <see cref="BundleUnlockRateLimiter"/> over a fleet-wide shared
|
||||
/// window (register row 52). This is the overload the production composition root uses.
|
||||
/// </summary>
|
||||
/// <param name="timeProvider">Clock used for the process-local fallback window.</param>
|
||||
/// <param name="window">Trailing window over which attempts are counted (typically 1 hour).</param>
|
||||
/// <param name="sharedWindow">Shared cross-node attempt window; when null the limiter falls back to process-local counters.</param>
|
||||
/// <param name="logger">Optional logger. Used for the fail-open Warning when the shared window is unavailable.</param>
|
||||
public BundleUnlockRateLimiter(
|
||||
TimeProvider timeProvider,
|
||||
TimeSpan window,
|
||||
IBundleUnlockAttemptWindow? sharedWindow,
|
||||
ILogger<BundleUnlockRateLimiter>? logger)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(timeProvider);
|
||||
if (window <= TimeSpan.Zero)
|
||||
@@ -64,13 +95,81 @@ public sealed class BundleUnlockRateLimiter
|
||||
|
||||
_timeProvider = timeProvider;
|
||||
_window = window;
|
||||
_sharedWindow = sharedWindow;
|
||||
_logger = logger ?? NullLogger<BundleUnlockRateLimiter>.Instance;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Attempts to register a new passphrase try against the configured per-key
|
||||
/// limit. Returns <c>true</c> when the attempt is permitted (and recorded);
|
||||
/// returns <c>false</c> when the key has exhausted its budget for the trailing
|
||||
/// window — the caller should reject the unlock request with a 429-equivalent.
|
||||
/// Gets a value indicating whether this limiter is backed by a fleet-wide shared
|
||||
/// window. False means the cap degrades to per-node, per-process counters.
|
||||
/// </summary>
|
||||
public bool IsFleetWide => _sharedWindow is not null;
|
||||
|
||||
/// <summary>
|
||||
/// Registers a passphrase try against the configured per-key limit, using the
|
||||
/// fleet-wide shared window when one is registered and the process-local window
|
||||
/// otherwise. This is the method the importer calls.
|
||||
/// </summary>
|
||||
/// <param name="clientKey">Opaque caller identifier — typically the remote IP.</param>
|
||||
/// <param name="maxAttemptsPerWindow">The trailing-window cap (<c>MaxUnlockAttemptsPerIpPerHour</c>).</param>
|
||||
/// <param name="ct">Cancellation token.</param>
|
||||
/// <returns>
|
||||
/// <c>true</c> if the attempt is permitted; <c>false</c> if the caller has exhausted
|
||||
/// its budget. Also <c>true</c> when the shared window is unavailable — see the
|
||||
/// class remarks on why the throttle fails open.
|
||||
/// </returns>
|
||||
public async Task<bool> TryRegisterAttemptAsync(
|
||||
string clientKey, int maxAttemptsPerWindow, CancellationToken ct = default)
|
||||
{
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(clientKey);
|
||||
if (maxAttemptsPerWindow < 1)
|
||||
{
|
||||
throw new ArgumentOutOfRangeException(
|
||||
nameof(maxAttemptsPerWindow), "Limit must be at least 1.");
|
||||
}
|
||||
|
||||
if (_sharedWindow is null)
|
||||
{
|
||||
return TryRegisterAttempt(clientKey, maxAttemptsPerWindow);
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
return await _sharedWindow
|
||||
.TryRegisterAttemptAsync(clientKey, maxAttemptsPerWindow, _window, ct)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
catch (OperationCanceledException) when (ct.IsCancellationRequested)
|
||||
{
|
||||
// The caller's own cancellation, not a store failure — do not swallow it.
|
||||
throw;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
// Fail OPEN: a best-effort rate cap must never take the import surface down
|
||||
// with the database. The Warning is the operator's signal that the fleet cap
|
||||
// is currently not being enforced.
|
||||
_logger.LogWarning(
|
||||
ex,
|
||||
"Fleet-wide bundle-unlock throttle unavailable; permitting the unlock attempt for client "
|
||||
+ "{ClientKey} without enforcing the {Cap}/hour cap.",
|
||||
clientKey, maxAttemptsPerWindow);
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Attempts to register a new passphrase try against the configured per-key limit in
|
||||
/// the PROCESS-LOCAL window. Returns <c>true</c> when the attempt is permitted (and
|
||||
/// recorded); returns <c>false</c> when the key has exhausted its budget for the
|
||||
/// trailing window — the caller should reject the unlock request with a 429-equivalent.
|
||||
/// <para>
|
||||
/// Register row 52: this bypasses the fleet-wide window. Callers on the unlock path
|
||||
/// must use <see cref="TryRegisterAttemptAsync"/>, which routes to the shared window
|
||||
/// when one is registered and falls back here otherwise. This overload remains public
|
||||
/// for hosts that register Transport without a shared window, and for the tests that
|
||||
/// pin the sliding-window arithmetic directly.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
/// <param name="clientKey">
|
||||
/// Opaque caller identifier — typically the remote IP, but any stable per-source
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
namespace ZB.MOM.WW.ScadaBridge.Transport.Import;
|
||||
|
||||
/// <summary>
|
||||
/// Register row 52: a sliding window of bundle-unlock attempts that is shared across the
|
||||
/// central pair, so <c>TransportOptions.MaxUnlockAttemptsPerIpPerHour</c> is a FLEET cap
|
||||
/// rather than a per-node, per-process one.
|
||||
/// <para>
|
||||
/// Implementations are expected to THROW on store failure. The fail-open decision belongs
|
||||
/// to <see cref="BundleUnlockRateLimiter"/>, which owns the policy (log a Warning and
|
||||
/// permit the attempt), not to the storage layer.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public interface IBundleUnlockAttemptWindow
|
||||
{
|
||||
/// <summary>
|
||||
/// Registers one unlock attempt against the shared window and reports whether it was
|
||||
/// within budget. Implementations must NOT record an attempt they are rejecting —
|
||||
/// a denial burns no budget, matching the in-memory limiter's contract (register
|
||||
/// row 44), so a client that keeps hammering cannot extend its own lockout.
|
||||
/// </summary>
|
||||
/// <param name="clientKey">Throttle bucket — the caller's remote IP, or the shared unattributed bucket.</param>
|
||||
/// <param name="maxAttemptsPerWindow">The trailing-window cap (<c>MaxUnlockAttemptsPerIpPerHour</c>).</param>
|
||||
/// <param name="window">Trailing window the attempts are counted over (one hour in production).</param>
|
||||
/// <param name="ct">Cancellation token.</param>
|
||||
/// <returns><c>true</c> when the attempt was recorded (within budget); <c>false</c> when the bucket is exhausted.</returns>
|
||||
Task<bool> TryRegisterAttemptAsync(
|
||||
string clientKey, int maxAttemptsPerWindow, TimeSpan window, CancellationToken ct = default);
|
||||
}
|
||||
@@ -0,0 +1,167 @@
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Transport.Import;
|
||||
|
||||
/// <summary>
|
||||
/// Register row 52: <see cref="IBundleUnlockAttemptWindow"/> backed by the central MS SQL
|
||||
/// <c>BundleUnlockAttempt</c> table — the HA store both central nodes already share, so
|
||||
/// the fleet-wide cap needs no new infrastructure, no new port and no consensus protocol.
|
||||
/// <para>
|
||||
/// Registered as a singleton (the limiter that consumes it is one) and therefore resolves
|
||||
/// a scoped <see cref="IBundleUnlockAttemptRepository"/> from its own
|
||||
/// <see cref="IServiceScopeFactory"/> scope per call, the same way the central purge
|
||||
/// singletons reach the database.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Accuracy is deliberately approximate.</b> The count-then-insert pair is not atomic
|
||||
/// across the two central nodes, so genuinely simultaneous attempts can overshoot the cap
|
||||
/// by up to the number in flight. A conditional single-statement insert would close that,
|
||||
/// at the cost of provider-specific SQL for a best-effort rate cap layered over an
|
||||
/// already-authenticated, role-gated surface. Overshoot is bounded and self-corrects
|
||||
/// within the window; it is not worth raw SQL.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public sealed class SqlBundleUnlockAttemptWindow : IBundleUnlockAttemptWindow
|
||||
{
|
||||
/// <summary>
|
||||
/// Minimum wall-clock gap between opportunistic prunes. The prune runs on the unlock
|
||||
/// path (Transport owns no central scheduler, and the design constraint was "no new
|
||||
/// background service"), so it is claimed at most this often per process — the same
|
||||
/// rate-limited opportunistic-sweep shape <c>DeploymentService</c> uses for terminal
|
||||
/// deployment records.
|
||||
/// </summary>
|
||||
internal static readonly TimeSpan PruneMinInterval = TimeSpan.FromMinutes(15);
|
||||
|
||||
/// <summary>
|
||||
/// Upper bound on rows deleted by one opportunistic prune pass. Bounded so a sweep on
|
||||
/// a request path can never become a long-running delete, even if the table grew while
|
||||
/// prunes were failing.
|
||||
/// </summary>
|
||||
internal const int PruneBatchSize = 1000;
|
||||
|
||||
/// <summary>
|
||||
/// Multiple of the trailing window that rows are retained for before becoming
|
||||
/// prunable. Kept above 1 so a prune racing a count can never delete a row the count
|
||||
/// still needs.
|
||||
/// </summary>
|
||||
internal const int PruneRetentionWindows = 2;
|
||||
|
||||
private readonly IServiceScopeFactory _scopeFactory;
|
||||
private readonly TimeProvider _timeProvider;
|
||||
private readonly ILogger<SqlBundleUnlockAttemptWindow> _logger;
|
||||
|
||||
/// <summary>Ticks of the last claimed prune, as a CAS-guarded process-local claim.</summary>
|
||||
private long _lastPruneTicks;
|
||||
|
||||
/// <summary>
|
||||
/// Initializes a new <see cref="SqlBundleUnlockAttemptWindow"/>.
|
||||
/// </summary>
|
||||
/// <param name="scopeFactory">Scope factory used to resolve the scoped attempt repository per call.</param>
|
||||
/// <param name="timeProvider">Clock used for the window boundary and the prune cadence.</param>
|
||||
/// <param name="logger">Optional logger for prune diagnostics.</param>
|
||||
public SqlBundleUnlockAttemptWindow(
|
||||
IServiceScopeFactory scopeFactory,
|
||||
TimeProvider timeProvider,
|
||||
ILogger<SqlBundleUnlockAttemptWindow>? logger = null)
|
||||
{
|
||||
_scopeFactory = scopeFactory ?? throw new ArgumentNullException(nameof(scopeFactory));
|
||||
_timeProvider = timeProvider ?? throw new ArgumentNullException(nameof(timeProvider));
|
||||
_logger = logger ?? NullLogger<SqlBundleUnlockAttemptWindow>.Instance;
|
||||
|
||||
// Seed the prune claim to "now" so a freshly started node does not prune on its
|
||||
// very first unlock — the table is almost certainly already tidy, and a node that
|
||||
// restarts under load would otherwise prune on every restart.
|
||||
_lastPruneTicks = _timeProvider.GetUtcNow().UtcTicks;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task<bool> TryRegisterAttemptAsync(
|
||||
string clientKey, int maxAttemptsPerWindow, TimeSpan window, CancellationToken ct = default)
|
||||
{
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(clientKey);
|
||||
if (maxAttemptsPerWindow < 1)
|
||||
{
|
||||
throw new ArgumentOutOfRangeException(
|
||||
nameof(maxAttemptsPerWindow), "Limit must be at least 1.");
|
||||
}
|
||||
|
||||
if (window <= TimeSpan.Zero)
|
||||
{
|
||||
throw new ArgumentOutOfRangeException(nameof(window), "Window must be positive.");
|
||||
}
|
||||
|
||||
var now = _timeProvider.GetUtcNow().UtcDateTime;
|
||||
var key = clientKey.Trim();
|
||||
|
||||
using var scope = _scopeFactory.CreateScope();
|
||||
var repo = scope.ServiceProvider.GetRequiredService<IBundleUnlockAttemptRepository>();
|
||||
|
||||
// Two round trips rather than one conditional insert — see the class remarks on
|
||||
// why the resulting overshoot is accepted. Exceptions propagate: the fail-open
|
||||
// decision lives in BundleUnlockRateLimiter.
|
||||
var used = await repo.CountAttemptsSinceAsync(key, now - window, ct).ConfigureAwait(false);
|
||||
if (used >= maxAttemptsPerWindow)
|
||||
{
|
||||
// Rejected attempts are NOT recorded — a denial burns no budget, so a client
|
||||
// hammering the endpoint cannot roll its own window forward indefinitely.
|
||||
return false;
|
||||
}
|
||||
|
||||
await repo.RecordAttemptAsync(key, now, ct).ConfigureAwait(false);
|
||||
|
||||
await TryPruneAsync(repo, now, window, ct).ConfigureAwait(false);
|
||||
return true;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Opportunistic, rate-limited, bounded prune of aged-out attempt rows. Best-effort:
|
||||
/// any failure is logged and swallowed, because retention maintenance must never fail
|
||||
/// the operator's actual unlock.
|
||||
/// </summary>
|
||||
/// <param name="repo">Repository resolved for the current call's scope.</param>
|
||||
/// <param name="nowUtc">Current UTC instant.</param>
|
||||
/// <param name="window">The trailing window; rows older than a multiple of it are prunable.</param>
|
||||
/// <param name="ct">Cancellation token.</param>
|
||||
/// <returns>A task that completes when the prune attempt has finished (or been skipped).</returns>
|
||||
private async Task TryPruneAsync(
|
||||
IBundleUnlockAttemptRepository repo, DateTime nowUtc, TimeSpan window, CancellationToken ct)
|
||||
{
|
||||
var previous = Interlocked.Read(ref _lastPruneTicks);
|
||||
if (nowUtc.Ticks - previous < PruneMinInterval.Ticks)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
// Claim the sweep so concurrent unlocks on this node don't all prune at once.
|
||||
if (Interlocked.CompareExchange(ref _lastPruneTicks, nowUtc.Ticks, previous) != previous)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
var cutoff = nowUtc - (window * PruneRetentionWindows);
|
||||
var deleted = await repo
|
||||
.PruneAttemptsOlderThanAsync(cutoff, PruneBatchSize, ct)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
if (deleted > 0)
|
||||
{
|
||||
_logger.LogDebug(
|
||||
"Pruned {Deleted} bundle-unlock attempt row(s) older than {Cutoff:o}.", deleted, cutoff);
|
||||
}
|
||||
}
|
||||
catch (OperationCanceledException) when (ct.IsCancellationRequested)
|
||||
{
|
||||
throw;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
// Never fail the unlock for a retention sweep.
|
||||
_logger.LogWarning(ex, "Opportunistic bundle-unlock attempt prune failed; will retry on a later attempt.");
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,5 +1,6 @@
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.DependencyInjection.Extensions;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Transport;
|
||||
using ZB.MOM.WW.ScadaBridge.TemplateEngine.Validation;
|
||||
@@ -43,10 +44,22 @@ public static class ServiceCollectionExtensions
|
||||
services.AddScoped<DependencyResolver>();
|
||||
services.AddScoped<IBundleExporter, BundleExporter>();
|
||||
services.AddSingleton<IBundleSessionStore, BundleSessionStore>();
|
||||
// T-004: per-IP-per-hour unlock rate limiter — design doc §11. Singleton
|
||||
// so the trailing-hour window is shared across every importer scope; the
|
||||
// counters live in-memory and reset on host restart (by design).
|
||||
services.AddSingleton<BundleUnlockRateLimiter>();
|
||||
// T-004: per-IP-per-hour unlock rate limiter — design doc §11. Singleton so the
|
||||
// trailing-hour window is shared across every importer scope.
|
||||
//
|
||||
// Register row 52: the window is FLEET-WIDE, backed by the central MS SQL
|
||||
// BundleUnlockAttempt table that both central nodes already share — the counters
|
||||
// used to be per-node and per-process, so the effective cap was N x limit per
|
||||
// client and a restart cleared the window. The window resolves its scoped
|
||||
// repository from its own scope per call, so it is safe as a singleton.
|
||||
// Constructed explicitly rather than by convention so it is unambiguous which
|
||||
// constructor overload production uses.
|
||||
services.AddSingleton<IBundleUnlockAttemptWindow, SqlBundleUnlockAttemptWindow>();
|
||||
services.AddSingleton(sp => new BundleUnlockRateLimiter(
|
||||
sp.GetRequiredService<TimeProvider>(),
|
||||
BundleUnlockRateLimiter.DefaultWindow,
|
||||
sp.GetRequiredService<IBundleUnlockAttemptWindow>(),
|
||||
sp.GetService<ILogger<BundleUnlockRateLimiter>>()));
|
||||
// T-007: periodic eviction sweep so abandoned sessions clear without
|
||||
// needing a fresh Get() to trigger lazy eviction.
|
||||
services.AddHostedService<BundleSessionEvictionService>();
|
||||
|
||||
+90
@@ -0,0 +1,90 @@
|
||||
using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Repositories;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests.Repositories;
|
||||
|
||||
/// <summary>
|
||||
/// Register row 52 coverage for <see cref="BundleUnlockAttemptRepository"/> over the
|
||||
/// in-memory SQLite harness — the fleet-wide bundle-unlock throttle window's persistence
|
||||
/// layer. Exercises the windowed count (including the bucket-isolation and window-boundary
|
||||
/// cases the throttle depends on) and the bounded prune.
|
||||
/// </summary>
|
||||
public class BundleUnlockAttemptRepositoryTests
|
||||
{
|
||||
// Fixed UTC base so the window assertions are deterministic.
|
||||
private static readonly DateTime Base = new(2026, 6, 1, 12, 0, 0, DateTimeKind.Utc);
|
||||
|
||||
private static ScadaBridgeDbContext NewContext() => SqliteTestHelper.CreateInMemoryContext();
|
||||
|
||||
[Fact]
|
||||
public async Task CountAttemptsSinceAsync_CountsOnlyTheGivenBucketInsideTheWindow()
|
||||
{
|
||||
await using var ctx = NewContext();
|
||||
var repo = new BundleUnlockAttemptRepository(ctx);
|
||||
|
||||
// Two inside the trailing hour, one aged out, one from a different client.
|
||||
await repo.RecordAttemptAsync("10.0.0.1", Base.AddMinutes(-10));
|
||||
await repo.RecordAttemptAsync("10.0.0.1", Base.AddMinutes(-50));
|
||||
await repo.RecordAttemptAsync("10.0.0.1", Base.AddMinutes(-90));
|
||||
await repo.RecordAttemptAsync("10.0.0.2", Base.AddMinutes(-5));
|
||||
|
||||
var count = await repo.CountAttemptsSinceAsync("10.0.0.1", Base.AddHours(-1));
|
||||
|
||||
Assert.Equal(2, count);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task CountAttemptsSinceAsync_IsInclusiveAtTheWindowFloor()
|
||||
{
|
||||
// The floor is inclusive, matching the in-memory limiter's prune predicate
|
||||
// (entries at exactly the cutoff still count). Pinned because an off-by-one here
|
||||
// would silently hand every client one extra attempt per window.
|
||||
await using var ctx = NewContext();
|
||||
var repo = new BundleUnlockAttemptRepository(ctx);
|
||||
|
||||
await repo.RecordAttemptAsync("10.0.0.1", Base.AddHours(-1));
|
||||
|
||||
Assert.Equal(1, await repo.CountAttemptsSinceAsync("10.0.0.1", Base.AddHours(-1)));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task CountAttemptsSinceAsync_UnknownBucket_ReturnsZero()
|
||||
{
|
||||
await using var ctx = NewContext();
|
||||
var repo = new BundleUnlockAttemptRepository(ctx);
|
||||
|
||||
Assert.Equal(0, await repo.CountAttemptsSinceAsync("<unattributed>", Base.AddHours(-1)));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task PruneAttemptsOlderThanAsync_DeletesOnlyAgedRows_AndIsBounded()
|
||||
{
|
||||
await using var ctx = NewContext();
|
||||
var repo = new BundleUnlockAttemptRepository(ctx);
|
||||
|
||||
for (var i = 0; i < 5; i++)
|
||||
{
|
||||
await repo.RecordAttemptAsync("10.0.0.1", Base.AddHours(-5).AddMinutes(i));
|
||||
}
|
||||
|
||||
await repo.RecordAttemptAsync("10.0.0.1", Base.AddMinutes(-1));
|
||||
|
||||
// Bounded: only 3 of the 5 eligible rows go in this pass.
|
||||
var deleted = await repo.PruneAttemptsOlderThanAsync(Base.AddHours(-2), maxRows: 3);
|
||||
Assert.Equal(3, deleted);
|
||||
|
||||
// The recent row is untouched, and a second pass clears the remainder.
|
||||
Assert.Equal(1, await repo.CountAttemptsSinceAsync("10.0.0.1", Base.AddHours(-1)));
|
||||
Assert.Equal(2, await repo.PruneAttemptsOlderThanAsync(Base.AddHours(-2), maxRows: 3));
|
||||
Assert.Equal(0, await repo.PruneAttemptsOlderThanAsync(Base.AddHours(-2), maxRows: 3));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task PruneAttemptsOlderThanAsync_RejectsANonPositiveBatchSize()
|
||||
{
|
||||
await using var ctx = NewContext();
|
||||
var repo = new BundleUnlockAttemptRepository(ctx);
|
||||
|
||||
await Assert.ThrowsAsync<ArgumentOutOfRangeException>(
|
||||
() => repo.PruneAttemptsOlderThanAsync(Base, maxRows: 0));
|
||||
}
|
||||
}
|
||||
@@ -3219,6 +3219,51 @@ public class ManagementActorTests : TestKit, IDisposable
|
||||
Assert.Contains("Too many bundle unlock attempts", response.Error);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ImportBundleCommand_LockedBundle_SurfacesTheCuratedReason()
|
||||
{
|
||||
// Register row 51: the per-bundle 3-strike lockout used to reach the CLI as
|
||||
// MapFault's generic "An internal error occurred (CorrelationId=...)" because
|
||||
// BundleLockedException lived in Transport, which ManagementService does not
|
||||
// reference. It now lives in Commons alongside BundleUnlockThrottledException
|
||||
// and is curated the same way.
|
||||
var (_, importer) = AddBundleSubstitutes();
|
||||
|
||||
importer.LoadAsync(Arg.Any<Stream>(), Arg.Any<string?>(), Arg.Any<CancellationToken>(), Arg.Any<string?>())
|
||||
.Returns<Commons.Types.Transport.BundleSession>(_ =>
|
||||
throw new Commons.Types.Transport.BundleLockedException("abc123", 3));
|
||||
|
||||
var payload = Convert.ToBase64String(new byte[] { 0x01, 0x02, 0x03 });
|
||||
var actor = CreateActor();
|
||||
actor.Tell(Envelope(new ImportBundleCommand(payload, "wrong", "skip"), "Administrator"));
|
||||
|
||||
var response = ExpectMsg<ManagementError>(TimeSpan.FromSeconds(5));
|
||||
Assert.Equal("COMMAND_FAILED", response.ErrorCode);
|
||||
Assert.Contains("locked after 3 failed unlock attempts", response.Error);
|
||||
Assert.DoesNotContain("An internal error occurred", response.Error);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void PreviewBundleCommand_LockedBundle_SurfacesTheCuratedReason()
|
||||
{
|
||||
// Register row 51: preview shares the same LoadAsync fault surface as import,
|
||||
// so it needs the same curated arm.
|
||||
var (_, importer) = AddBundleSubstitutes();
|
||||
|
||||
importer.LoadAsync(Arg.Any<Stream>(), Arg.Any<string?>(), Arg.Any<CancellationToken>(), Arg.Any<string?>())
|
||||
.Returns<Commons.Types.Transport.BundleSession>(_ =>
|
||||
throw new Commons.Types.Transport.BundleLockedException("abc123", 3));
|
||||
|
||||
var payload = Convert.ToBase64String(new byte[] { 0x01, 0x02, 0x03 });
|
||||
var actor = CreateActor();
|
||||
actor.Tell(Envelope(new PreviewBundleCommand(payload, "wrong"), "Administrator"));
|
||||
|
||||
var response = ExpectMsg<ManagementError>(TimeSpan.FromSeconds(5));
|
||||
Assert.Equal("COMMAND_FAILED", response.ErrorCode);
|
||||
Assert.Contains("locked after 3 failed unlock attempts", response.Error);
|
||||
Assert.DoesNotContain("An internal error occurred", response.Error);
|
||||
}
|
||||
|
||||
// ========================================================================
|
||||
// Native alarm source CRUD (Task 21)
|
||||
// ========================================================================
|
||||
|
||||
+263
@@ -0,0 +1,263 @@
|
||||
using Microsoft.Data.Sqlite;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using NSubstitute;
|
||||
using NSubstitute.ExceptionExtensions;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
|
||||
using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase;
|
||||
using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Repositories;
|
||||
using ZB.MOM.WW.ScadaBridge.Transport.Import;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Transport.IntegrationTests.Import;
|
||||
|
||||
/// <summary>
|
||||
/// Register row 52: the bundle-unlock throttle's counters used to be per-node AND
|
||||
/// per-process, so the effective fleet cap was N x <c>MaxUnlockAttemptsPerIpPerHour</c>
|
||||
/// per client and a node restart silently cleared the window. These tests stand up TWO
|
||||
/// independent limiter instances — the stand-in for central-a and central-b — over ONE
|
||||
/// shared relational store and prove they enforce a single combined window, plus the
|
||||
/// fail-open behaviour a store outage must produce.
|
||||
/// </summary>
|
||||
public sealed class FleetWideUnlockThrottleTests : IDisposable
|
||||
{
|
||||
private readonly SqliteConnection _connection;
|
||||
private readonly ServiceProvider _provider;
|
||||
|
||||
public FleetWideUnlockThrottleTests()
|
||||
{
|
||||
// One open in-memory SQLite connection for the fixture's lifetime — the database
|
||||
// lives exactly as long as the connection, and both "nodes" resolve scoped
|
||||
// contexts over it, which is what makes the store genuinely shared.
|
||||
_connection = new SqliteConnection("DataSource=:memory:");
|
||||
_connection.Open();
|
||||
|
||||
var services = new ServiceCollection();
|
||||
services.AddDbContext<ScadaBridgeDbContext>(opts => opts.UseSqlite(_connection));
|
||||
services.AddScoped<IBundleUnlockAttemptRepository, BundleUnlockAttemptRepository>();
|
||||
_provider = services.BuildServiceProvider();
|
||||
|
||||
using var scope = _provider.CreateScope();
|
||||
scope.ServiceProvider.GetRequiredService<ScadaBridgeDbContext>().Database.EnsureCreated();
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
_provider.Dispose();
|
||||
_connection.Dispose();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Builds a limiter over the shared store, standing in for one central node's
|
||||
/// singleton. Each call returns a fully independent instance (its own process-local
|
||||
/// fallback buckets, its own prune claim) — exactly the isolation the two real nodes
|
||||
/// have.
|
||||
/// </summary>
|
||||
/// <param name="clock">Clock the node observes.</param>
|
||||
/// <returns>A limiter whose window is the shared store.</returns>
|
||||
private BundleUnlockRateLimiter NewNode(TimeProvider clock) => new(
|
||||
clock,
|
||||
BundleUnlockRateLimiter.DefaultWindow,
|
||||
new SqlBundleUnlockAttemptWindow(
|
||||
_provider.GetRequiredService<IServiceScopeFactory>(), clock),
|
||||
logger: null);
|
||||
|
||||
[Fact]
|
||||
public async Task TwoNodesSharingOneStore_EnforceASingleCombinedWindow()
|
||||
{
|
||||
// The headline row-52 contract: attempts spread across both central nodes count
|
||||
// against ONE budget. Before this, five attempts per node were both allowed.
|
||||
var clock = new TestClock(new DateTimeOffset(2026, 6, 1, 12, 0, 0, TimeSpan.Zero));
|
||||
var nodeA = NewNode(clock);
|
||||
var nodeB = NewNode(clock);
|
||||
|
||||
for (var i = 0; i < 5; i++)
|
||||
{
|
||||
Assert.True(await nodeA.TryRegisterAttemptAsync("10.0.0.1", 10));
|
||||
Assert.True(await nodeB.TryRegisterAttemptAsync("10.0.0.1", 10));
|
||||
}
|
||||
|
||||
// The combined budget of 10 is now spent — on EITHER node.
|
||||
Assert.False(await nodeA.TryRegisterAttemptAsync("10.0.0.1", 10));
|
||||
Assert.False(await nodeB.TryRegisterAttemptAsync("10.0.0.1", 10));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task DeniedAttempts_BurnNoBudget()
|
||||
{
|
||||
// Register row 44's contract, preserved through the move to the shared store: a
|
||||
// rejected attempt is not recorded, so hammering cannot roll the window forward
|
||||
// and extend the client's own lockout.
|
||||
var clock = new TestClock(new DateTimeOffset(2026, 6, 1, 12, 0, 0, TimeSpan.Zero));
|
||||
var nodeA = NewNode(clock);
|
||||
|
||||
for (var i = 0; i < 3; i++)
|
||||
{
|
||||
Assert.True(await nodeA.TryRegisterAttemptAsync("10.0.0.1", 3));
|
||||
}
|
||||
|
||||
for (var i = 0; i < 5; i++)
|
||||
{
|
||||
Assert.False(await nodeA.TryRegisterAttemptAsync("10.0.0.1", 3));
|
||||
}
|
||||
|
||||
// Exactly the 3 permitted attempts were persisted — the 5 denials added nothing.
|
||||
using var scope = _provider.CreateScope();
|
||||
var repo = scope.ServiceProvider.GetRequiredService<IBundleUnlockAttemptRepository>();
|
||||
var recorded = await repo.CountAttemptsSinceAsync(
|
||||
"10.0.0.1", clock.GetUtcNow().UtcDateTime.AddHours(-1));
|
||||
Assert.Equal(3, recorded);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task TheWindowIsRestartDurable()
|
||||
{
|
||||
// A node restart used to clear the window outright. A brand-new limiter instance
|
||||
// (the restarted process) must still see the budget the old one spent.
|
||||
var clock = new TestClock(new DateTimeOffset(2026, 6, 1, 12, 0, 0, TimeSpan.Zero));
|
||||
var before = NewNode(clock);
|
||||
|
||||
for (var i = 0; i < 3; i++)
|
||||
{
|
||||
Assert.True(await before.TryRegisterAttemptAsync("10.0.0.1", 3));
|
||||
}
|
||||
|
||||
var afterRestart = NewNode(clock);
|
||||
Assert.False(await afterRestart.TryRegisterAttemptAsync("10.0.0.1", 3));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task TheWindowSlides_SoAnAgedOutBudgetIsReleased()
|
||||
{
|
||||
var clock = new TestClock(new DateTimeOffset(2026, 6, 1, 12, 0, 0, TimeSpan.Zero));
|
||||
var node = NewNode(clock);
|
||||
|
||||
for (var i = 0; i < 3; i++)
|
||||
{
|
||||
Assert.True(await node.TryRegisterAttemptAsync("10.0.0.1", 3));
|
||||
}
|
||||
|
||||
Assert.False(await node.TryRegisterAttemptAsync("10.0.0.1", 3));
|
||||
|
||||
clock.Advance(TimeSpan.FromMinutes(61));
|
||||
Assert.True(await node.TryRegisterAttemptAsync("10.0.0.1", 3));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task BucketsAreIsolatedPerClient()
|
||||
{
|
||||
var clock = new TestClock(new DateTimeOffset(2026, 6, 1, 12, 0, 0, TimeSpan.Zero));
|
||||
var node = NewNode(clock);
|
||||
|
||||
Assert.True(await node.TryRegisterAttemptAsync("10.0.0.1", 1));
|
||||
Assert.False(await node.TryRegisterAttemptAsync("10.0.0.1", 1));
|
||||
|
||||
// A different client — and the shared unattributed bucket — are unaffected.
|
||||
Assert.True(await node.TryRegisterAttemptAsync("10.0.0.2", 1));
|
||||
Assert.True(await node.TryRegisterAttemptAsync("<unattributed>", 1));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task StoreFailure_FailsOpen_AndDoesNotBlockTheUnlock()
|
||||
{
|
||||
// Availability of legitimate imports outranks a best-effort rate cap: a database
|
||||
// error must permit the attempt, not take the import surface down with the DB.
|
||||
var window = Substitute.For<IBundleUnlockAttemptWindow>();
|
||||
window
|
||||
.TryRegisterAttemptAsync(
|
||||
Arg.Any<string>(), Arg.Any<int>(), Arg.Any<TimeSpan>(), Arg.Any<CancellationToken>())
|
||||
.ThrowsAsync(new InvalidOperationException("database unavailable"));
|
||||
|
||||
var limiter = new BundleUnlockRateLimiter(
|
||||
TimeProvider.System, BundleUnlockRateLimiter.DefaultWindow, window, logger: null);
|
||||
|
||||
for (var i = 0; i < 20; i++)
|
||||
{
|
||||
Assert.True(await limiter.TryRegisterAttemptAsync("10.0.0.1", 1));
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task CallerCancellation_IsNotSwallowedByTheFailOpenPath()
|
||||
{
|
||||
// Fail-open must not hide the caller's own cancellation — that would turn an
|
||||
// aborted request into a silently permitted attempt.
|
||||
var window = Substitute.For<IBundleUnlockAttemptWindow>();
|
||||
window
|
||||
.TryRegisterAttemptAsync(
|
||||
Arg.Any<string>(), Arg.Any<int>(), Arg.Any<TimeSpan>(), Arg.Any<CancellationToken>())
|
||||
.ThrowsAsync(new OperationCanceledException());
|
||||
|
||||
var limiter = new BundleUnlockRateLimiter(
|
||||
TimeProvider.System, BundleUnlockRateLimiter.DefaultWindow, window, logger: null);
|
||||
|
||||
using var cts = new CancellationTokenSource();
|
||||
await cts.CancelAsync();
|
||||
|
||||
await Assert.ThrowsAnyAsync<OperationCanceledException>(
|
||||
() => limiter.TryRegisterAttemptAsync("10.0.0.1", 10, cts.Token));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task WithoutASharedWindow_TheLimiterStillEnforcesAProcessLocalCap()
|
||||
{
|
||||
// Hosts that register Transport without a shared window keep the pre-row-52
|
||||
// behaviour rather than losing the cap entirely.
|
||||
var clock = new TestClock(new DateTimeOffset(2026, 6, 1, 12, 0, 0, TimeSpan.Zero));
|
||||
var limiter = new BundleUnlockRateLimiter(clock, BundleUnlockRateLimiter.DefaultWindow);
|
||||
|
||||
Assert.False(limiter.IsFleetWide);
|
||||
Assert.True(await limiter.TryRegisterAttemptAsync("10.0.0.1", 1));
|
||||
Assert.False(await limiter.TryRegisterAttemptAsync("10.0.0.1", 1));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task AgedRowsArePrunedOpportunistically_OnceThePruneIntervalHasElapsed()
|
||||
{
|
||||
// No background service: the prune rides the unlock path, claimed at most once per
|
||||
// PruneMinInterval. Rows are retained for PruneRetentionWindows x window so a prune
|
||||
// racing a count can never delete a row the count still needs.
|
||||
var clock = new TestClock(new DateTimeOffset(2026, 6, 1, 12, 0, 0, TimeSpan.Zero));
|
||||
var node = NewNode(clock);
|
||||
|
||||
Assert.True(await node.TryRegisterAttemptAsync("10.0.0.1", 10));
|
||||
|
||||
// Move past both the retention horizon (2 hours) and the prune cadence.
|
||||
clock.Advance(TimeSpan.FromHours(3));
|
||||
Assert.True(await node.TryRegisterAttemptAsync("10.0.0.2", 10));
|
||||
|
||||
using var scope = _provider.CreateScope();
|
||||
var ctx = scope.ServiceProvider.GetRequiredService<ScadaBridgeDbContext>();
|
||||
var remaining = await ctx.BundleUnlockAttempts.AsNoTracking().ToListAsync();
|
||||
|
||||
// The aged row is gone; the one just written by the pruning attempt survives.
|
||||
Assert.Equal("10.0.0.2", Assert.Single(remaining).ClientKey);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task RecentRowsSurviveAPrunePass()
|
||||
{
|
||||
// Negative control for the prune horizon: an attempt still inside the retention
|
||||
// window must not be deleted, or the throttle would forget live budget.
|
||||
var clock = new TestClock(new DateTimeOffset(2026, 6, 1, 12, 0, 0, TimeSpan.Zero));
|
||||
var node = NewNode(clock);
|
||||
|
||||
Assert.True(await node.TryRegisterAttemptAsync("10.0.0.1", 10));
|
||||
|
||||
// Past the prune cadence (15 min) but inside the 2-hour retention horizon.
|
||||
clock.Advance(TimeSpan.FromMinutes(30));
|
||||
Assert.True(await node.TryRegisterAttemptAsync("10.0.0.2", 10));
|
||||
|
||||
using var scope = _provider.CreateScope();
|
||||
var ctx = scope.ServiceProvider.GetRequiredService<ScadaBridgeDbContext>();
|
||||
Assert.Equal(2, await ctx.BundleUnlockAttempts.AsNoTracking().CountAsync());
|
||||
}
|
||||
|
||||
/// <summary>Minimal mutable-clock <see cref="TimeProvider"/> for deterministic windows.</summary>
|
||||
private sealed class TestClock : TimeProvider
|
||||
{
|
||||
private DateTimeOffset _now;
|
||||
public TestClock(DateTimeOffset start) { _now = start; }
|
||||
public override DateTimeOffset GetUtcNow() => _now;
|
||||
public void Advance(TimeSpan by) => _now = _now.Add(by);
|
||||
}
|
||||
}
|
||||
+248
@@ -0,0 +1,248 @@
|
||||
using System.Diagnostics.Metrics;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Observability;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Transport;
|
||||
using ZB.MOM.WW.ScadaBridge.Transport.Import;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Transport.Tests.Import;
|
||||
|
||||
/// <summary>
|
||||
/// Register row 53: <c>MaxConcurrentImportSessions</c> had no observability — nothing
|
||||
/// counted open sessions and nothing logged a rejection at the cap, so the register-row-43
|
||||
/// session leak could only be inferred from rotating test failures. These tests pin the
|
||||
/// two signals that close that gap: the
|
||||
/// <c>scadabridge.central.transport.import_sessions.open</c> gauge and the Warning-level
|
||||
/// log emitted when <see cref="BundleSessionStore.Open"/> rejects at the cap.
|
||||
/// </summary>
|
||||
public sealed class BundleSessionObservabilityTests
|
||||
{
|
||||
private const string OpenSessionsGauge = "scadabridge.central.transport.import_sessions.open";
|
||||
|
||||
// ------------------------------------------------------------------
|
||||
// Warning log on cap rejection
|
||||
// ------------------------------------------------------------------
|
||||
|
||||
[Fact]
|
||||
public void Open_AtCap_LogsWarningWithCountAndCap()
|
||||
{
|
||||
// The rejection used to be visible only as the exception the caller saw. The
|
||||
// operator-facing signal must name BOTH the current count and the cap — that
|
||||
// pair is what distinguishes genuine concurrency from a leak.
|
||||
var clock = new TestClock(DateTimeOffset.UtcNow);
|
||||
var opts = Options.Create(new TransportOptions { MaxConcurrentImportSessions = 3 });
|
||||
var logger = new CapturingLogger();
|
||||
var sut = new BundleSessionStore(opts, clock, logger);
|
||||
var future = clock.GetUtcNow().AddMinutes(30);
|
||||
|
||||
for (var i = 0; i < 3; i++)
|
||||
{
|
||||
sut.Open(SessionExpiringAt(future));
|
||||
}
|
||||
|
||||
Assert.Throws<InvalidOperationException>(() => sut.Open(SessionExpiringAt(future)));
|
||||
|
||||
var warning = Assert.Single(logger.Entries, e => e.Level == LogLevel.Warning);
|
||||
Assert.Contains("Bundle import session rejected", warning.Message, StringComparison.Ordinal);
|
||||
Assert.Contains("3 of 3", warning.Message, StringComparison.Ordinal);
|
||||
Assert.Contains("MaxConcurrentImportSessions", warning.Message, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Open_UnderCap_LogsNothing()
|
||||
{
|
||||
// Negative control: the Warning must be a cap signal, not per-open noise.
|
||||
var clock = new TestClock(DateTimeOffset.UtcNow);
|
||||
var opts = Options.Create(new TransportOptions { MaxConcurrentImportSessions = 3 });
|
||||
var logger = new CapturingLogger();
|
||||
var sut = new BundleSessionStore(opts, clock, logger);
|
||||
|
||||
sut.Open(SessionExpiringAt(clock.GetUtcNow().AddMinutes(30)));
|
||||
|
||||
Assert.Empty(logger.Entries);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Open_AtCapOfExpiredSessions_ReclaimsAndLogsNothing()
|
||||
{
|
||||
// A store that is full only of stale entries reclaims and admits the caller —
|
||||
// that is not a cap hit and must not produce a Warning, or the signal would fire
|
||||
// routinely on any node whose sweep has not yet run.
|
||||
var clock = new TestClock(DateTimeOffset.UtcNow);
|
||||
var opts = Options.Create(new TransportOptions { MaxConcurrentImportSessions = 2 });
|
||||
var logger = new CapturingLogger();
|
||||
var sut = new BundleSessionStore(opts, clock, logger);
|
||||
|
||||
sut.Open(SessionExpiringAt(clock.GetUtcNow().AddMinutes(5)));
|
||||
sut.Open(SessionExpiringAt(clock.GetUtcNow().AddMinutes(5)));
|
||||
clock.Advance(TimeSpan.FromMinutes(10));
|
||||
|
||||
sut.Open(SessionExpiringAt(clock.GetUtcNow().AddMinutes(30)));
|
||||
|
||||
Assert.Empty(logger.Entries);
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------
|
||||
// OpenSessionCount / gauge
|
||||
// ------------------------------------------------------------------
|
||||
|
||||
[Fact]
|
||||
public void OpenSessionCount_ExcludesExpiredButUnsweptSessions()
|
||||
{
|
||||
// The eviction sweep runs once a minute, so a raw dictionary count would report
|
||||
// the store as pinned at the cap for up to a sweep interval after a burst had
|
||||
// actually drained — exactly the false signal this instrument exists to rule out.
|
||||
var clock = new TestClock(DateTimeOffset.UtcNow);
|
||||
var sut = new BundleSessionStore(Options.Create(new TransportOptions()), clock, NullLogger<BundleSessionStore>.Instance);
|
||||
|
||||
sut.Open(SessionExpiringAt(clock.GetUtcNow().AddMinutes(5)));
|
||||
var live = SessionExpiringAt(clock.GetUtcNow().AddMinutes(60));
|
||||
sut.Open(live);
|
||||
|
||||
Assert.Equal(2, sut.OpenSessionCount);
|
||||
|
||||
clock.Advance(TimeSpan.FromMinutes(10));
|
||||
|
||||
// The short session has expired but nothing has swept it yet.
|
||||
Assert.Equal(1, sut.OpenSessionCount);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task EvictionService_RegistersTheGaugeProvider_AndClearsItOnStop()
|
||||
{
|
||||
// The eviction service is Transport's only lifecycle seam, so it owns the
|
||||
// process-global gauge slot. The gauge is read as a delta against a baseline
|
||||
// because the instrument is a process-wide static.
|
||||
var clock = new TestClock(DateTimeOffset.UtcNow);
|
||||
var store = new BundleSessionStore(Options.Create(new TransportOptions()), clock, NullLogger<BundleSessionStore>.Instance);
|
||||
var service = new BundleSessionEvictionService(store, NullLogger<BundleSessionEvictionService>.Instance);
|
||||
|
||||
var baseline = ReadGauge();
|
||||
|
||||
await service.StartAsync(CancellationToken.None);
|
||||
try
|
||||
{
|
||||
store.Open(SessionExpiringAt(clock.GetUtcNow().AddMinutes(30)));
|
||||
store.Open(SessionExpiringAt(clock.GetUtcNow().AddMinutes(30)));
|
||||
|
||||
Assert.Equal(2, ReadGauge());
|
||||
}
|
||||
finally
|
||||
{
|
||||
await service.StopAsync(CancellationToken.None);
|
||||
}
|
||||
|
||||
// After the clear the gauge falls back to reporting 0, not through a dead store.
|
||||
Assert.Equal(0, ReadGauge());
|
||||
Assert.Equal(0, baseline);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task EvictionService_StopDoesNotClobberANewerInstancesProvider()
|
||||
{
|
||||
// Identity-checked compare-and-clear: a late stop of a superseded instance must
|
||||
// leave a newer registrant's provider in place (mirrors the StoreAndForward
|
||||
// queue-depth gauge's contract).
|
||||
var clock = new TestClock(DateTimeOffset.UtcNow);
|
||||
var oldStore = new BundleSessionStore(Options.Create(new TransportOptions()), clock, NullLogger<BundleSessionStore>.Instance);
|
||||
var newStore = new BundleSessionStore(Options.Create(new TransportOptions()), clock, NullLogger<BundleSessionStore>.Instance);
|
||||
var oldService = new BundleSessionEvictionService(oldStore, NullLogger<BundleSessionEvictionService>.Instance);
|
||||
var newService = new BundleSessionEvictionService(newStore, NullLogger<BundleSessionEvictionService>.Instance);
|
||||
|
||||
await oldService.StartAsync(CancellationToken.None);
|
||||
await newService.StartAsync(CancellationToken.None);
|
||||
try
|
||||
{
|
||||
newStore.Open(SessionExpiringAt(clock.GetUtcNow().AddMinutes(30)));
|
||||
|
||||
// Late stop of the superseded instance.
|
||||
await oldService.StopAsync(CancellationToken.None);
|
||||
|
||||
Assert.Equal(1, ReadGauge());
|
||||
}
|
||||
finally
|
||||
{
|
||||
await newService.StopAsync(CancellationToken.None);
|
||||
}
|
||||
|
||||
Assert.Equal(0, ReadGauge());
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------
|
||||
// Helpers
|
||||
// ------------------------------------------------------------------
|
||||
|
||||
/// <summary>
|
||||
/// Forces one synchronous observation of the open-import-sessions gauge and returns
|
||||
/// the measured value. Mirrors the MeterListener helper used by the Communication
|
||||
/// and StoreAndForward telemetry suites.
|
||||
/// </summary>
|
||||
/// <returns>The gauge's current value, or 0 if it never reported.</returns>
|
||||
private static long ReadGauge()
|
||||
{
|
||||
long observed = 0;
|
||||
using var listener = new MeterListener();
|
||||
listener.InstrumentPublished = (instrument, l) =>
|
||||
{
|
||||
if (instrument.Meter.Name == ScadaBridgeTelemetry.MeterName
|
||||
&& instrument.Name == OpenSessionsGauge)
|
||||
{
|
||||
l.EnableMeasurementEvents(instrument);
|
||||
}
|
||||
};
|
||||
listener.SetMeasurementEventCallback<long>((_, measurement, _, _) => observed = measurement);
|
||||
listener.Start();
|
||||
listener.RecordObservableInstruments();
|
||||
return observed;
|
||||
}
|
||||
|
||||
private static BundleSession SessionExpiringAt(DateTimeOffset expiresAt) => new()
|
||||
{
|
||||
SessionId = Guid.NewGuid(),
|
||||
Manifest = StubManifest(),
|
||||
DecryptedContent = Array.Empty<byte>(),
|
||||
ExpiresAt = expiresAt,
|
||||
};
|
||||
|
||||
private static BundleManifest StubManifest() => new(
|
||||
BundleFormatVersion: 1,
|
||||
SchemaVersion: "1.0",
|
||||
CreatedAtUtc: DateTimeOffset.UnixEpoch,
|
||||
SourceEnvironment: "test",
|
||||
ExportedBy: "t",
|
||||
ScadaBridgeVersion: "1",
|
||||
ContentHash: "0",
|
||||
Encryption: null,
|
||||
Summary: new BundleSummary(0, 0, 0, 0, 0, 0, 0, 0),
|
||||
Contents: Array.Empty<ManifestContentEntry>());
|
||||
|
||||
/// <summary>Minimal mutable-clock <see cref="TimeProvider"/> for deterministic expiry.</summary>
|
||||
private sealed class TestClock : TimeProvider
|
||||
{
|
||||
private DateTimeOffset _now;
|
||||
public TestClock(DateTimeOffset start) { _now = start; }
|
||||
public override DateTimeOffset GetUtcNow() => _now;
|
||||
public void Advance(TimeSpan by) => _now = _now.Add(by);
|
||||
}
|
||||
|
||||
/// <summary>Captures formatted log entries so the cap-rejection Warning can be asserted.</summary>
|
||||
private sealed class CapturingLogger : ILogger<BundleSessionStore>
|
||||
{
|
||||
public List<(LogLevel Level, string Message)> Entries { get; } = [];
|
||||
|
||||
public IDisposable? BeginScope<TState>(TState state) where TState : notnull => null;
|
||||
|
||||
public bool IsEnabled(LogLevel logLevel) => true;
|
||||
|
||||
public void Log<TState>(
|
||||
LogLevel logLevel,
|
||||
EventId eventId,
|
||||
TState state,
|
||||
Exception? exception,
|
||||
Func<TState, Exception?, string> formatter)
|
||||
{
|
||||
Entries.Add((logLevel, formatter(state, exception)));
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user