merge: SQL driver health review findings resolved (Driver.Sql-004/-005/-006/-008/-010)
# Conflicts: # docs/drivers/Sql.md
This commit is contained in:
@@ -105,13 +105,13 @@ RED evidence: with the shipped `OracleDialect` (1000-entry `IN` limit) and 1001
|
||||
| Severity | Low |
|
||||
| Category | Error handling & resilience |
|
||||
| Location | `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql/SqlDriver.cs:651` |
|
||||
| Status | Open |
|
||||
| Status | Resolved |
|
||||
|
||||
**Description:** `HandlePollError` skips the connection class by testing `ex is DbException`, on the stated premise that "a subscribed-read outage surfaces as a DbException the reader threw, which ReadAsync has ALREADY classified". Not every unreachable-database throw is a `DbException`: `SqlPollReader.QueryAsync` raises `InvalidOperationException` when the provider factory returns no connection, and a provider may raise `ArgumentException`/`FormatException` from a malformed connection string on the same open path. For those, `ReadAsync` writes the good endpoint-bearing `LastError` and transitions the host to `Stopped`, rethrows, and the poll engine then routes the same exception here — where `Degrade` overwrites that message with the guidance-free "poll failed: TypeName" and the outage is logged a second time. The comment describes the intended behaviour; the type test does not implement it.
|
||||
|
||||
**Recommendation:** Mark the exception as already-classified at the one place that knows (e.g. `ReadAsync` wrapping the throw, or a per-call flag) rather than inferring it from the exception type, so the reader's richer `LastError` survives.
|
||||
|
||||
**Resolution:** _(open)_
|
||||
**Resolution (2026-08-10):** Fixed — `90f3016a`. `ReadAsync`'s catch now records the exception *instance* in a `_classifiedByRead` field before rethrowing, and `HandlePollError` defers only when `Interlocked.CompareExchange` matches that same object (consume-once); the `ex is DbException` type test is gone. A field rather than `Exception.Data` or a wrapper: `Data` is provider-owned and may be read-only, and wrapping would change the type `IReadable`'s callers and the poll engine see. TDD: `HandlePollError_forANonDbExceptionReadAsyncClassified_stillDefers_soTheGoodLastErrorSurvives` drives a `BreakableFactory` that hands back no connection (the reader's `InvalidOperationException` path) and was RED — the endpoint-bearing `LastError` was overwritten with "poll failed: InvalidOperationException". Its falsifiability control, `HandlePollError_forAnExceptionReadAsyncNeverSaw_degrades_evenWhenItIsADbException`, was RED against the type test too (a `DbException` the engine raised was silently swallowed), and pins that the marker is per-instance rather than per-type. The pre-existing DbException test was rewritten to drive the real ReadAsync→HandlePollError round trip instead of handing `HandlePollError` a bare exception.
|
||||
|
||||
---
|
||||
|
||||
@@ -122,13 +122,13 @@ RED evidence: with the shipped `OracleDialect` (1000-entry `IN` limit) and 1001
|
||||
| Severity | Low |
|
||||
| Category | Correctness & logic bugs |
|
||||
| Location | `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql/SqlDriver.cs:641` |
|
||||
| Status | Open |
|
||||
| Status | Resolved |
|
||||
|
||||
**Description:** `IsConnectionClass` treats `BadCommunicationError` as "the database did not answer", so a poll carrying it reports the endpoint `Stopped`. But that code is minted by `GroupResult.Failed`, which the reader raises only after the connection has opened — its own comment says "the connection opened, so the server is up — this query is what failed (an identifier that is not in the catalog, a lock, a permission)". A single group failing on a lock, a revoked `SELECT`, a schema change or an over-long `IN` list (see Driver.Sql-002) therefore reports the whole database endpoint as down. The two types disagree about what the same status code means.
|
||||
|
||||
**Recommendation:** Either restrict the connection class to `BadTimeout` (a wedged socket is the case the host-state signal is for), or give a post-open query failure its own status so the two classifications cannot drift. Note the host state is currently only consumed through `IHostConnectivityProbe`, which is unwired (Gitea #521) — the operator-visible half today is the `LastError` wording and the Degraded verdict.
|
||||
|
||||
**Resolution:** _(open)_
|
||||
**Resolution (2026-08-10):** Fixed — `ee1f1b28`. `IsConnectionClass` is now `BadTimeout` only, and `BadCommunicationError` moves to a new `IsQueryFailureClass` with its own branch in `ObservePollOutcome`: still Degraded (every tag in that group is Bad), but the host stays **Running** and the `LastError` points at the table/column, the grant and any lock rather than at the network. The published status code is unchanged — the drift was in what the driver *read into* it, not in what the reader emits, so the wire behaviour and the `GroupResult.Failed` contract stay put. The first recommendation was rejected on its own: restricting the class without adding the branch would have left an all-groups-failing driver reporting Healthy. TDD: `ReadAsync_whenTheQueryFailsAfterTheConnectionOpened_degradesButLeavesTheEndpointUp` drops the fixture's table out from under a driver whose catalog gate has already passed, and was RED (`LastError` blamed communication; the host went `Stopped`). `ReadAsync_whenEveryTagTimesOut_degradesHealthAndReportsTheHostStopped` is the standing control that a genuine no-answer outage still reports `Stopped`.
|
||||
|
||||
---
|
||||
|
||||
@@ -139,13 +139,13 @@ RED evidence: with the shipped `OracleDialect` (1000-entry `IN` limit) and 1001
|
||||
| Severity | Low |
|
||||
| Category | Correctness & logic bugs |
|
||||
| Location | `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql/SqlDriver.cs:599` |
|
||||
| Status | Open |
|
||||
| Status | Resolved |
|
||||
|
||||
**Description:** `ObservePollOutcome` advances `LastSuccessfulRead` only when at least one snapshot is Good. Several codes are proof the database answered without being Good: `BadNoData` (the query ran, the row is absent), `Uncertain`/`Bad` (a present row with a NULL cell) and `BadTypeMismatch` (a cell came back and would not coerce). A driver whose every tag reads one of those — a truncated staging table, a key set that was deleted — leaves `LastSuccessfulRead` frozen. That is benign on its own (health is correctly left unchanged, because these are authoring/data facts), but it interacts with the comms-loss watchdog: once anything has degraded the driver, the conjunction "non-Healthy AND `LastSuccessfulRead` frozen for 90 s" (CLAUDE.md, Gitea #527) trips against a database that is answering every poll on time, and the node force-reconnects a healthy connection.
|
||||
|
||||
**Recommendation:** Advance `LastSuccessfulRead` on evidence-of-answer codes, not only on Good — the honest reading of "the database answered" the watchdog was designed around — while leaving the health state classification exactly as it is.
|
||||
|
||||
**Resolution:** _(open)_
|
||||
**Resolution (2026-08-10):** Fixed — `a0a49d33`. A named `IsAnswerEvidence` predicate now decides the clock, separately from the health verdict: `Good`, `BadNoData`, `Uncertain`, `Bad`, `BadTypeMismatch` and `BadCommunicationError` (post-Driver.Sql-005 that one means the connection opened and the server refused the query — so the transport works and a forced reconnect could not fix it). Deliberately excluded, and each pinned by a control: `BadTimeout` (the database did not answer) and `BadNodeIdUnknown` / `BadConfigurationError` / `BadInternalError` (minted before any statement was executed — a blanket "not a timeout" rule would keep the clock fresh on a driver that never reached the database, disabling the watchdog rather than fixing it). The health *state* classification is untouched: an answered-but-not-Good poll advances the clock and reports the host Running, and changes nothing else. TDD: `ReadAsync_whenTheDatabaseAnswersWithNoRow_advancesTheEvidenceClock_withoutChangingHealth` (degrade first, then read a deleted key) was RED with a frozen clock, as were four of the six `ObservePollOutcome_everyEvidenceOfAnswerCode_advancesTheEvidenceClock` theory cases; the two negative theories and `ReadAsync_anUnresolvableRef_isNoEvidenceTheDatabaseAnswered_soTheEvidenceClockStands` were green throughout and are the falsifiability controls.
|
||||
|
||||
---
|
||||
|
||||
@@ -173,13 +173,13 @@ RED evidence: with the shipped `OracleDialect` (1000-entry `IN` limit) and 1001
|
||||
| Severity | Low |
|
||||
| Category | Documentation & comments |
|
||||
| Location | `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql/SqlPollReader.cs:556` |
|
||||
| Status | Open |
|
||||
| Status | Resolved |
|
||||
|
||||
**Description:** `SelectWideRow`'s remarks justify giving the `topByTimestamp` form the dialect's single-row limit on the grounds that "there the `ORDER BY` makes 'the first row' a defined answer". That holds only when the ordering column is unique. `ORDER BY ts DESC` plus a one-row limit over a column with ties returns an arbitrary row among the tied ones, and — unlike the where-pair form, whose whole argument for not taking the limit is that `Rows.Count > 1` is the evidence of ambiguity — the reader can never detect it, because the server has already discarded the other rows. The browse-commit default makes the tie shape reachable rather than theoretical: `SqlBrowseSession.PreferredTimestampColumn` will happily nominate a `date`-typed column (see Driver.Sql.Browser-001), which ties for every row written on the same day.
|
||||
|
||||
**Recommendation:** State the tie caveat where the claim is made, and in `docs/drivers/Sql.md` alongside the `topByTimestamp` example: the model requires a selector that identifies one row, and a non-unique ordering column silently does not.
|
||||
|
||||
**Resolution:** _(open)_
|
||||
**Resolution (2026-08-10):** Fixed — `9d4760fd`, **documentation only**. `SelectWideRow`'s over-claim ("the `ORDER BY` makes 'the first row' a defined answer") now reads that the `ORDER BY` fixes only *which* rows can be first, followed by an explicit caveat: with a non-unique ordering column the server returns an arbitrary row among the tied ones and this method can never detect it, because the discarded rows never reach it — there is no `Rows.Count > 1` left to warn on, unlike the where-pair form. `docs/drivers/Sql.md` carries the same warning beside the `topByTimestamp` example, naming the reachable shape (a `date`-typed column ties for every row written on the same day, and browse-commit will nominate one) and the fix (a column unique per row, or a latest-row view). Behaviour deliberately unchanged: the entry argues for the doc, and the only behavioural remedy — a tie-break column in the emitted `ORDER BY` — lives in `SqlGroupPlanner`, which is out of scope here and would need an authored notion of what breaks the tie.
|
||||
|
||||
---
|
||||
|
||||
@@ -207,10 +207,10 @@ RED evidence: with the shipped `OracleDialect` (1000-entry `IN` limit) and 1001
|
||||
| Severity | Low |
|
||||
| Category | Concurrency & thread safety |
|
||||
| Location | `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql/SqlDriver.cs:675` |
|
||||
| Status | Open |
|
||||
| Status | Resolved |
|
||||
|
||||
**Description:** Every health transition is a read-modify-write over a `Volatile`-accessed field with no compare-and-swap: `Degrade`, `TouchLastSuccessfulRead` and `SetHealthUnlessFaulted` each `ReadHealth()`, build a new record and `WriteHealth()` it. Concurrent poll loops (one per subscription, plus an ad-hoc `ReadAsync`) can interleave between the read and the write, so a `LastSuccessfulRead` advance can be lost, or a `Faulted` verdict recorded by a concurrent reinitialize can be overwritten by a stale non-Faulted snapshot despite the guard — the guard reads the field before the write rather than atomically with it. Impact is limited to the health surface (a stale timestamp or state until the next poll), which is why this is Low.
|
||||
|
||||
**Recommendation:** Use `Interlocked.CompareExchange` in a small retry loop for the read-modify-write helpers, or funnel health mutation through a lock — the `_probeLock` the host-state transition already uses would do.
|
||||
|
||||
**Resolution:** _(open)_
|
||||
**Resolution (2026-08-10):** Fixed — `336e79ea`. Every poll-driven transition now goes through one `UpdateHealth(Func<DriverHealth, DriverHealth?>)` compare-and-swap retry loop: `Degrade` and the new `RecordHealthyRead` via `UpdateHealthUnlessFaulted` (so the Faulted guard is evaluated against the *same* snapshot the CAS swaps out, closing the "guard reads before the write" half), `TouchLastSuccessfulRead` directly. Two details are load-bearing. The transform is re-run per attempt and must derive its timestamp from the clock rather than from the call site — the Healthy write previously computed `DateTime.UtcNow` before the call, which a retry would republish *stale* and rewind the clock; that is why the Healthy path became `RecordHealthyRead` instead of passing a prebuilt record. And `WriteHealth` survives, unconditional, for the Initialize/Shutdown lifecycle writes that are last-writer-wins by definition — a Volatile write racing a CAS loop only makes the loop retry and re-read, which is strictly safer than before. TDD: `HealthTransitions_fromConcurrentPolls_neverRewindTheEvidenceClock` runs an advancing and a degrading `ObservePollOutcome` loop against one driver while an observer watches `LastSuccessfulRead`; monotonicity is the observable a lost update violates (a `Degrade` that read the clock at t0 republishes it after an advance to t1). RED 3/3 runs before the fix, green 3/3 after.
|
||||
|
||||
@@ -80,6 +80,14 @@ the nested form.
|
||||
}
|
||||
```
|
||||
|
||||
⚠️ **`topByTimestamp` needs an ordering column that is unique per row.** It is executed as an
|
||||
`ORDER BY <column> DESC` with the dialect's single-row limit, so when the column ties the server
|
||||
returns an arbitrary row among the tied ones — and the driver cannot warn about it, because the other
|
||||
rows were discarded before the result set reached it (the `whereColumn`/`whereValue` form *does*
|
||||
warn, on `Rows.Count > 1`). A `date`-typed column ties for every row written on the same day, and
|
||||
browse-commit will nominate one. Prefer a `datetime2`/sequence column that is unique per row, or
|
||||
point the tag at a latest-row view.
|
||||
|
||||
Both value-bearing fields — `whereValue` here and `keyValue` on the `KeyValue` model — may be
|
||||
authored as a JSON string, number or boolean, and are captured as text. They are **bound as
|
||||
parameters** either way, and matched back as text (the key cell is stringified), so a key column
|
||||
|
||||
@@ -155,6 +155,20 @@ public sealed class SqlDriver
|
||||
|
||||
private DriverHealth _health = new(DriverState.Unknown, null, null);
|
||||
|
||||
/// <summary>
|
||||
/// The exception instance <see cref="ReadAsync"/> has already turned into a health verdict, held only
|
||||
/// until <see cref="HandlePollError"/> — which the poll engine calls with that same object — has
|
||||
/// consumed it (Driver.Sql-004).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// A field rather than a flag on the exception itself: <see cref="Exception.Data"/> is provider-owned
|
||||
/// and may be read-only, and wrapping the throw would change the type the engine and
|
||||
/// <see cref="IReadable"/>'s callers see. Concurrent poll loops can overwrite each other's mark; the
|
||||
/// loser is then merely re-degraded with a poorer message, which is the pre-fix behaviour rather than
|
||||
/// a new failure.
|
||||
/// </remarks>
|
||||
private Exception? _classifiedByRead;
|
||||
|
||||
/// <summary>Occurs when a subscribed tag's value or quality changes.</summary>
|
||||
public event EventHandler<DataChangeEventArgs>? OnDataChange;
|
||||
|
||||
@@ -528,6 +542,10 @@ public sealed class SqlDriver
|
||||
// on the log sink via the LogWarning exception parameter above.
|
||||
Degrade($"Sql driver '{_driverInstanceId}' could not reach {Endpoint}: {ex.GetType().Name}.");
|
||||
TransitionHostTo(HostState.Stopped);
|
||||
// Mark THIS exception object as already classified before it goes back to the poll engine, which
|
||||
// routes it to HandlePollError next. The mark is what makes the deferral there exact — see that
|
||||
// method for why inferring it from the exception's type was not (Driver.Sql-004).
|
||||
Volatile.Write(ref _classifiedByRead, ex);
|
||||
throw;
|
||||
}
|
||||
}
|
||||
@@ -577,11 +595,16 @@ public sealed class SqlDriver
|
||||
/// <summary>
|
||||
/// Classifies one poll's snapshots into a health verdict and a host state.
|
||||
/// <list type="bullet">
|
||||
/// <item>Any Good snapshot ⇒ the database answered: <c>LastSuccessfulRead</c> advances and the
|
||||
/// host is Running.</item>
|
||||
/// <item>Any <b>connection-class</b> Bad snapshot (<see cref="SqlStatusCodes.BadTimeout"/> /
|
||||
/// <see cref="SqlStatusCodes.BadCommunicationError"/>) ⇒ Degraded; when nothing at all was Good,
|
||||
/// the host is Stopped.</item>
|
||||
/// <item>Any Good snapshot ⇒ Healthy and the host is Running.</item>
|
||||
/// <item>Any snapshot that is <b>evidence the database answered</b>
|
||||
/// (<see cref="IsAnswerEvidence"/> — Good, but also an absent row, a NULL cell, an uncoercible
|
||||
/// cell) ⇒ <c>LastSuccessfulRead</c> advances and the host is Running, <em>without</em> changing
|
||||
/// the health state. That clock is what the comms-loss watchdog reads (Driver.Sql-006).</item>
|
||||
/// <item>Any <b>connection-class</b> Bad snapshot (<see cref="SqlStatusCodes.BadTimeout"/>) ⇒
|
||||
/// Degraded; when nothing at all answered, the host is Stopped.</item>
|
||||
/// <item>Any <b>query-failure</b> Bad snapshot (<see cref="SqlStatusCodes.BadCommunicationError"/>)
|
||||
/// ⇒ Degraded, but the host stays Running. See <see cref="IsQueryFailureClass"/>: that code is
|
||||
/// minted only after the connection opened, so it is evidence the endpoint is up.</item>
|
||||
/// <item>Bad codes that are <b>authoring</b> facts — an unresolvable RawPath, an absent row, a
|
||||
/// type mismatch, a rejected definition — change nothing. A tag typo must never report the
|
||||
/// database as down and send an operator to the wrong system.</item>
|
||||
@@ -602,18 +625,22 @@ public sealed class SqlDriver
|
||||
|
||||
var good = 0;
|
||||
var unreachable = 0;
|
||||
var queryFailed = 0;
|
||||
var answered = 0;
|
||||
foreach (var snapshot in snapshots)
|
||||
{
|
||||
if (SqlStatusCodes.IsGood(snapshot.StatusCode)) good++;
|
||||
else if (IsConnectionClass(snapshot.StatusCode)) unreachable++;
|
||||
else if (IsQueryFailureClass(snapshot.StatusCode)) queryFailed++;
|
||||
|
||||
if (IsAnswerEvidence(snapshot.StatusCode)) answered++;
|
||||
}
|
||||
|
||||
if (unreachable > 0)
|
||||
{
|
||||
Degrade(
|
||||
$"{unreachable} of {snapshots.Count} Sql tag(s) timed out or failed communication against " +
|
||||
$"{Endpoint} on the last poll.");
|
||||
if (good > 0)
|
||||
$"{unreachable} of {snapshots.Count} Sql tag(s) timed out against {Endpoint} on the last poll.");
|
||||
if (answered > 0)
|
||||
{
|
||||
// Partial: something answered, so the endpoint is up — but the driver is not healthy.
|
||||
TouchLastSuccessfulRead();
|
||||
@@ -627,19 +654,92 @@ public sealed class SqlDriver
|
||||
return;
|
||||
}
|
||||
|
||||
if (good == 0) return; // authoring-class Bad only — not a statement about the database.
|
||||
if (queryFailed > 0)
|
||||
{
|
||||
// The connection opened and the server answered — with an error. Degraded (every tag in that
|
||||
// group is Bad) but emphatically NOT a reachability verdict: reporting the endpoint down over a
|
||||
// dropped table or a revoked SELECT points the operator at the wrong system, and the reconnect
|
||||
// it would eventually provoke cannot fix a query.
|
||||
Degrade(
|
||||
$"{queryFailed} of {snapshots.Count} Sql tag(s) failed their query against {Endpoint} on the " +
|
||||
"last poll; the connection opened, so check the table/column, the grant and any lock rather " +
|
||||
"than the network.");
|
||||
TouchLastSuccessfulRead();
|
||||
TransitionHostTo(HostState.Running);
|
||||
return;
|
||||
}
|
||||
|
||||
if (answered == 0) return; // authoring-class Bad only — not a statement about the database.
|
||||
|
||||
if (good == 0)
|
||||
{
|
||||
// The database answered, just not with a value: an absent row, a NULL cell, a cell that would
|
||||
// not coerce. Those are data/authoring facts, so the health STATE is deliberately left exactly
|
||||
// as it was — but they are still proof the transport works, and the evidence clock the
|
||||
// comms-loss watchdog reads must say so (Driver.Sql-006).
|
||||
TouchLastSuccessfulRead();
|
||||
TransitionHostTo(HostState.Running);
|
||||
return;
|
||||
}
|
||||
|
||||
// Route through the Faulted guard, not a raw WriteHealth: a poll still in flight when a concurrent
|
||||
// ReinitializeAsync has recorded Faulted (the engine's ~5 s teardown wait is shorter than the 15 s
|
||||
// default operationTimeout, so a stale poll can still complete afterwards) must not flip Faulted back
|
||||
// to Healthy with no real recovery. Only a successful (re)initialize clears Faulted.
|
||||
SetHealthUnlessFaulted(new DriverHealth(DriverState.Healthy, DateTime.UtcNow, null));
|
||||
RecordHealthyRead();
|
||||
TransitionHostTo(HostState.Running);
|
||||
}
|
||||
|
||||
/// <summary>True for the status codes that mean "the database did not answer", as opposed to "the data is not there".</summary>
|
||||
private static bool IsConnectionClass(uint statusCode)
|
||||
=> statusCode is SqlStatusCodes.BadTimeout or SqlStatusCodes.BadCommunicationError;
|
||||
/// <summary>
|
||||
/// True for the status codes that mean "the database did not answer", as opposed to "the data is not
|
||||
/// there" or "the query was refused".
|
||||
/// <para><see cref="SqlStatusCodes.BadCommunicationError"/> is deliberately <b>not</b> here despite
|
||||
/// its name (Driver.Sql-005): the reader mints it from <c>GroupResult.Failed</c>, which is reachable
|
||||
/// only after <c>OpenAsync</c> returned. Treating it as "did not answer" made one group failing on a
|
||||
/// schema change, a lock or a revoked grant report the whole endpoint down. See
|
||||
/// <see cref="IsQueryFailureClass"/>.</para>
|
||||
/// </summary>
|
||||
/// <param name="statusCode">The status code to classify.</param>
|
||||
/// <returns>True when the code means the database did not answer at all.</returns>
|
||||
private static bool IsConnectionClass(uint statusCode) => statusCode is SqlStatusCodes.BadTimeout;
|
||||
|
||||
/// <summary>
|
||||
/// True for the status codes that mean "the connection opened but the query failed" — a schema
|
||||
/// change, a lock, a missing grant, a statement the provider rejected. Degrading, but proof the
|
||||
/// endpoint is up.
|
||||
/// </summary>
|
||||
/// <param name="statusCode">The status code to classify.</param>
|
||||
/// <returns>True when the code means the server answered the connection and refused the query.</returns>
|
||||
private static bool IsQueryFailureClass(uint statusCode)
|
||||
=> statusCode is SqlStatusCodes.BadCommunicationError;
|
||||
|
||||
/// <summary>
|
||||
/// True for the status codes that are proof the database <b>answered</b> this poll — the evidence
|
||||
/// <c>LastSuccessfulRead</c> records, which is not the same question as "is this value good".
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>Driver.Sql-006. <c>DriverInstanceActor</c>'s comms-loss watchdog force-reconnects a driver
|
||||
/// that is non-Healthy <b>and</b> whose <c>LastSuccessfulRead</c> has been frozen for 90 s (Gitea
|
||||
/// #527), on the stated premise that the clock is the honest evidence the transport works. Advancing
|
||||
/// it on Good alone made a truncated staging table or a deleted key set look — to that watchdog —
|
||||
/// exactly like a dead socket, so a database answering every poll on time got its connection torn
|
||||
/// down and rebuilt every 90 s, forever.</para>
|
||||
/// <para>What is deliberately excluded is as load-bearing as what is included:
|
||||
/// <see cref="SqlStatusCodes.BadTimeout"/> is the database <em>not</em> answering, and
|
||||
/// <see cref="SqlStatusCodes.BadNodeIdUnknown"/> / <see cref="SqlStatusCodes.BadConfigurationError"/>
|
||||
/// / <see cref="SqlStatusCodes.BadInternalError"/> are minted before any statement is executed. A
|
||||
/// blanket "anything that is not a timeout" would keep the clock fresh on a driver that never once
|
||||
/// reached the database, which disables the watchdog rather than fixing it.</para>
|
||||
/// </remarks>
|
||||
/// <param name="statusCode">The status code to classify.</param>
|
||||
/// <returns>True when the code could only have been produced by a database that answered.</returns>
|
||||
private static bool IsAnswerEvidence(uint statusCode) => statusCode is
|
||||
SqlStatusCodes.Good // the row is there and readable
|
||||
or SqlStatusCodes.BadNoData // the query ran; that row is absent
|
||||
or SqlStatusCodes.Uncertain // a present row with a NULL cell (nullIsBad=false)
|
||||
or SqlStatusCodes.Bad // …the same row under nullIsBad=true
|
||||
or SqlStatusCodes.BadTypeMismatch // a cell came back and would not coerce
|
||||
or SqlStatusCodes.BadCommunicationError; // the connection opened; the server refused the query
|
||||
|
||||
/// <summary>
|
||||
/// Routes a poll-loop reader failure to the health surface (05/STAB-9). Deliberately does <b>not</b>
|
||||
@@ -650,11 +750,20 @@ public sealed class SqlDriver
|
||||
/// <param name="ex">The exception the poll engine caught.</param>
|
||||
internal void HandlePollError(Exception ex)
|
||||
{
|
||||
// A subscribed-read outage surfaces as a DbException the reader threw, which ReadAsync has ALREADY
|
||||
// classified — good, endpoint-bearing LastError + host Stopped — before rethrowing. The poll engine
|
||||
// then routes that same exception here. Re-degrading would overwrite the good LastError with a worse,
|
||||
// guidance-free one and log the outage a second time, so skip the connection class ReadAsync owns.
|
||||
if (ex is DbException) return;
|
||||
// A subscribed-read outage is classified by ReadAsync — good, endpoint-bearing LastError + host
|
||||
// Stopped — before it rethrows, and the poll engine then routes that SAME exception object here.
|
||||
// Re-degrading would overwrite the good LastError with a worse, guidance-free one and log the outage
|
||||
// a second time, so defer to the classification that already happened.
|
||||
//
|
||||
// Matched on the exception INSTANCE ReadAsync marked, not on its type. `ex is DbException` was the
|
||||
// stated intent — "the class ReadAsync owns" — but not what it implemented (Driver.Sql-004): the
|
||||
// reader raises a plain InvalidOperationException when the provider factory returns no connection,
|
||||
// and a provider can raise ArgumentException/FormatException off a malformed connection string on
|
||||
// the same open path. Those are classified by ReadAsync exactly like a DbException, yet a type test
|
||||
// let them through and clobbered its message; conversely a DbException that never went through
|
||||
// ReadAsync — the poll engine's own throw — was swallowed. Consumed once, so a later unclassified
|
||||
// exception is never mistaken for this one.
|
||||
if (ReferenceEquals(Interlocked.CompareExchange(ref _classifiedByRead, null, ex), ex)) return;
|
||||
|
||||
// Whatever reaches here — an engine contract-violation, or a non-DbException the provider raised
|
||||
// (e.g. an ArgumentException from a malformed connection string, which the parser echoes with
|
||||
@@ -673,36 +782,70 @@ public sealed class SqlDriver
|
||||
/// </summary>
|
||||
/// <param name="reason">The operator-facing reason, which must never carry the connection string.</param>
|
||||
private void Degrade(string reason)
|
||||
{
|
||||
var current = ReadHealth();
|
||||
SetHealthUnlessFaulted(new DriverHealth(DriverState.Degraded, current.LastSuccessfulRead, reason));
|
||||
}
|
||||
=> UpdateHealthUnlessFaulted(
|
||||
current => new DriverHealth(DriverState.Degraded, current.LastSuccessfulRead, reason));
|
||||
|
||||
/// <summary>
|
||||
/// Publishes <paramref name="value"/> unless the driver is already <see cref="DriverState.Faulted"/>
|
||||
/// — a liveness/config verdict that only a successful (re)initialize may clear. Every non-Initialize
|
||||
/// health write routes through here: <see cref="Degrade"/> and the poll's Healthy verdict in
|
||||
/// <see cref="ObservePollOutcome"/>, so a late in-flight poll cannot un-fault a driver a concurrent
|
||||
/// <see cref="ReinitializeAsync"/> just faulted.
|
||||
/// Records a poll that returned at least one Good value: Healthy, with the evidence clock advanced.
|
||||
/// <para>The timestamp is taken <b>inside</b> the transform, so a retried compare-and-swap publishes
|
||||
/// a fresh one rather than the one this call started with — which a concurrent advance could already
|
||||
/// have overtaken.</para>
|
||||
/// </summary>
|
||||
/// <param name="value">The health snapshot to publish when the driver is not Faulted.</param>
|
||||
private void SetHealthUnlessFaulted(DriverHealth value)
|
||||
{
|
||||
if (ReadHealth().State == DriverState.Faulted) return;
|
||||
WriteHealth(value);
|
||||
}
|
||||
private void RecordHealthyRead()
|
||||
=> UpdateHealthUnlessFaulted(
|
||||
static _ => new DriverHealth(DriverState.Healthy, DateTime.UtcNow, null));
|
||||
|
||||
/// <summary>Advances <c>LastSuccessfulRead</c> without changing the current state or error.</summary>
|
||||
private void TouchLastSuccessfulRead()
|
||||
=> UpdateHealth(static current => current with { LastSuccessfulRead = DateTime.UtcNow });
|
||||
|
||||
/// <summary>
|
||||
/// Applies <paramref name="project"/> unless the driver is already <see cref="DriverState.Faulted"/>
|
||||
/// — a liveness/config verdict that only a successful (re)initialize may clear. Every non-Initialize
|
||||
/// state write routes through here: <see cref="Degrade"/> and the poll's Healthy verdict in
|
||||
/// <see cref="ObservePollOutcome"/>, so a late in-flight poll cannot un-fault a driver a concurrent
|
||||
/// <see cref="ReinitializeAsync"/> just faulted. The guard is evaluated against the <em>same</em>
|
||||
/// snapshot the write then swaps out, so a Faulted verdict landing mid-update cannot be lost.
|
||||
/// </summary>
|
||||
/// <param name="project">Builds the snapshot to publish from the current one.</param>
|
||||
private void UpdateHealthUnlessFaulted(Func<DriverHealth, DriverHealth> project)
|
||||
=> UpdateHealth(current => current.State == DriverState.Faulted ? null : project(current));
|
||||
|
||||
/// <summary>
|
||||
/// The one read-modify-write of the health field, as a compare-and-swap retry loop.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Driver.Sql-010. Health is transitioned from every poll loop (one per subscription) plus any
|
||||
/// ad-hoc <see cref="ReadAsync"/>, and every transition needs the current snapshot to build the next
|
||||
/// one — <see cref="Degrade"/> preserves <c>LastSuccessfulRead</c>, the Faulted guard reads the state.
|
||||
/// A plain <c>Volatile</c> read followed by a <c>Volatile</c> write lets a concurrent update land in
|
||||
/// between and be silently overwritten: an advance of the evidence clock the comms-loss watchdog
|
||||
/// reads is lost (and the clock even moves backwards), or a Faulted verdict is downgraded despite the
|
||||
/// guard. <paramref name="transform"/> is re-run on each attempt, so it must be side-effect-free and
|
||||
/// must derive everything — timestamps included — from the snapshot it is handed or from the clock.
|
||||
/// Returning null declines the update.
|
||||
/// </remarks>
|
||||
/// <param name="transform">Builds the snapshot to publish, or returns null to leave health untouched.</param>
|
||||
private void UpdateHealth(Func<DriverHealth, DriverHealth?> transform)
|
||||
{
|
||||
var current = ReadHealth();
|
||||
WriteHealth(current with { LastSuccessfulRead = DateTime.UtcNow });
|
||||
while (true)
|
||||
{
|
||||
var current = Volatile.Read(ref _health);
|
||||
var next = transform(current);
|
||||
if (next is null) return;
|
||||
if (ReferenceEquals(Interlocked.CompareExchange(ref _health, next, current), current)) return;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Barrier-protected read of the multi-threaded health field.</summary>
|
||||
private DriverHealth ReadHealth() => Volatile.Read(ref _health);
|
||||
|
||||
/// <summary>Barrier-protected publish of a new health snapshot.</summary>
|
||||
/// <summary>
|
||||
/// Unconditional publish of a new health snapshot, for the lifecycle transitions that are
|
||||
/// last-writer-wins by definition (Initialize / Shutdown). Everything driven by a poll goes through
|
||||
/// <see cref="UpdateHealth"/> instead, because it has to read the current snapshot first.
|
||||
/// </summary>
|
||||
/// <param name="value">The health snapshot to publish.</param>
|
||||
private void WriteHealth(DriverHealth value) => Volatile.Write(ref _health, value);
|
||||
|
||||
/// <summary>Records a host-state change and raises <see cref="OnHostStatusChanged"/> — on transitions only.</summary>
|
||||
|
||||
@@ -566,9 +566,19 @@ public sealed class SqlPollReader
|
||||
/// one. Worse, it would destroy the only evidence this method has that the selector is ambiguous
|
||||
/// (<c>Rows.Count > 1</c>), converting a loud misconfiguration into a permanently silent one —
|
||||
/// precisely the failure mode this class's summary calls its top risk. The <c>topByTimestamp</c>
|
||||
/// form keeps its row limit because there the <c>ORDER BY</c> makes "the first row" a defined
|
||||
/// answer. The accepted cost is that an ambiguous selector materialises every matching row for one
|
||||
/// form keeps its row limit because there the <c>ORDER BY</c> at least fixes <em>which</em> rows can
|
||||
/// be first. The accepted cost is that an ambiguous selector materialises every matching row for one
|
||||
/// poll; the warning is what gets it fixed.</para>
|
||||
/// <para><b>⚠ The <c>topByTimestamp</c> answer is only defined when the ordering column is unique</b>
|
||||
/// (Driver.Sql-008). <c>ORDER BY ts DESC</c> plus a one-row limit over a column with ties returns an
|
||||
/// arbitrary row among the tied ones, and — unlike the where-pair form — this method can never
|
||||
/// detect it, because the server discarded the other rows before the result set was built: there is
|
||||
/// no <c>Rows.Count > 1</c> left to warn about. The tie shape is reachable rather than
|
||||
/// theoretical, since <c>SqlBrowseSession.PreferredTimestampColumn</c> will nominate a
|
||||
/// <c>date</c>-typed column, which ties for every row written on the same day. The model's premise is
|
||||
/// unchanged — the selector must identify one row — but for this form the operator, not the driver,
|
||||
/// is the only thing that can check it. Point the tag at a column that is unique per row (or at a
|
||||
/// latest-row view).</para>
|
||||
/// </summary>
|
||||
private object?[]? SelectWideRow(SqlQueryPlan plan, GroupResult outcome)
|
||||
{
|
||||
|
||||
@@ -224,6 +224,33 @@ public sealed class SqlDriverTests
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ReadAsync_whenTheQueryFailsAfterTheConnectionOpened_degradesButLeavesTheEndpointUp()
|
||||
{
|
||||
// Driver.Sql-005. BadCommunicationError is minted by GroupResult.Failed, which the reader raises only
|
||||
// AFTER the connection opened ("the connection opened, so the server is up — this query is what
|
||||
// failed"). Classifying it as connection-class reported the whole database endpoint Stopped over one
|
||||
// group failing on a dropped table, a revoked SELECT or an over-long IN list, sending the operator at
|
||||
// the network instead of at the schema. The driver must still degrade — every tag in that group is
|
||||
// Bad — but the endpoint is demonstrably up.
|
||||
using var fixture = new SqlitePollFixture();
|
||||
await using var driver = NewDriver(fixture, KvEntry("Speed", SqlitePollFixture.PresentKey));
|
||||
await driver.InitializeAsync(ConfigJson, CancellationToken.None);
|
||||
driver.GetHostStatuses()[0].State.ShouldBe(HostState.Running);
|
||||
|
||||
// The schema drifts out from under a driver whose catalog gate already passed.
|
||||
fixture.Execute($"DROP TABLE {SqlitePollFixture.KeyValueTable}");
|
||||
|
||||
var snapshots = await driver.ReadAsync(["Speed"], CancellationToken.None);
|
||||
|
||||
snapshots[0].StatusCode.ShouldBe(SqlStatusCodes.BadCommunicationError);
|
||||
var health = driver.GetHealth();
|
||||
health.State.ShouldBe(DriverState.Degraded);
|
||||
health.LastError.ShouldNotBeNull();
|
||||
health.LastError.ShouldContain("quer"); // names the query, not the network
|
||||
driver.GetHostStatuses()[0].State.ShouldBe(HostState.Running); // RED pre-fix: Stopped
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ReadAsync_anUnresolvableRef_isNotAConnectivityFact_soHealthAndHostStand()
|
||||
{
|
||||
@@ -242,6 +269,147 @@ public sealed class SqlDriverTests
|
||||
driver.GetHostStatuses()[0].State.ShouldBe(HostState.Running);
|
||||
}
|
||||
|
||||
// ---- the comms-loss watchdog's evidence clock (Driver.Sql-006) ----
|
||||
|
||||
[Fact]
|
||||
public async Task ReadAsync_whenTheDatabaseAnswersWithNoRow_advancesTheEvidenceClock_withoutChangingHealth()
|
||||
{
|
||||
// Driver.Sql-006. LastSuccessfulRead is what DriverInstanceActor's comms-loss watchdog reads: a
|
||||
// non-Healthy driver whose clock has been frozen for 90 s is force-reconnected (Gitea #527), on the
|
||||
// premise that the clock is the honest evidence the transport works. BadNoData IS that evidence —
|
||||
// the query ran and the row is absent — so a driver whose key set was deleted must not be
|
||||
// reconnected every 90 s against a database answering every poll on time. The health STATE stays
|
||||
// exactly as it was: an absent row is a data fact, not a connectivity one.
|
||||
using var fixture = new SqlitePollFixture();
|
||||
await using var driver = NewDriver(fixture, KvEntry("Missing", SqlitePollFixture.AbsentKey));
|
||||
await driver.InitializeAsync(ConfigJson, CancellationToken.None);
|
||||
|
||||
// Something has degraded the driver — the watchdog's first conjunct. Only then does the frozen clock
|
||||
// matter, which is why the scenario needs both.
|
||||
driver.HandlePollError(new InvalidOperationException("Reader contract violation: expected 1 snapshots"));
|
||||
var before = driver.GetHealth().LastSuccessfulRead.ShouldNotBeNull();
|
||||
await Task.Delay(20, TestContext.Current.CancellationToken);
|
||||
|
||||
var snapshots = await driver.ReadAsync(["Missing"], CancellationToken.None);
|
||||
|
||||
snapshots[0].StatusCode.ShouldBe(SqlStatusCodes.BadNoData);
|
||||
var health = driver.GetHealth();
|
||||
health.LastSuccessfulRead.ShouldNotBeNull().ShouldBeGreaterThan(before); // RED pre-fix: frozen
|
||||
health.State.ShouldBe(DriverState.Degraded); // classification untouched
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ReadAsync_anUnresolvableRef_isNoEvidenceTheDatabaseAnswered_soTheEvidenceClockStands()
|
||||
{
|
||||
// The falsifiability control: "advance on anything that is not Good" would pass the test above and
|
||||
// break the watchdog outright. BadNodeIdUnknown is minted by the resolver without any database
|
||||
// contact at all, so it proves nothing about the transport and must leave the clock alone —
|
||||
// otherwise a driver polling only typoed refs would look forever alive to the watchdog.
|
||||
using var fixture = new SqlitePollFixture();
|
||||
await using var driver = NewDriver(fixture, KvEntry("Speed", SqlitePollFixture.PresentKey));
|
||||
await driver.InitializeAsync(ConfigJson, CancellationToken.None);
|
||||
var before = driver.GetHealth().LastSuccessfulRead.ShouldNotBeNull();
|
||||
await Task.Delay(20, TestContext.Current.CancellationToken);
|
||||
|
||||
var snapshots = await driver.ReadAsync(["TypoedTagName"], CancellationToken.None);
|
||||
|
||||
snapshots[0].StatusCode.ShouldBe(SqlStatusCodes.BadNodeIdUnknown);
|
||||
driver.GetHealth().LastSuccessfulRead.ShouldBe(before);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(SqlStatusCodes.Good)] // the row is there and readable
|
||||
[InlineData(SqlStatusCodes.BadNoData)] // the query ran; the row is absent
|
||||
[InlineData(SqlStatusCodes.Uncertain)] // a present row with a NULL cell (nullIsBad=false)
|
||||
[InlineData(SqlStatusCodes.Bad)] // …the same row under nullIsBad=true
|
||||
[InlineData(SqlStatusCodes.BadTypeMismatch)] // a cell came back and would not coerce
|
||||
[InlineData(SqlStatusCodes.BadCommunicationError)] // the connection opened; the query was refused
|
||||
public async Task ObservePollOutcome_everyEvidenceOfAnswerCode_advancesTheEvidenceClock(uint statusCode)
|
||||
{
|
||||
using var fixture = new SqlitePollFixture();
|
||||
await using var driver = NewDriver(fixture, KvEntry("Speed", SqlitePollFixture.PresentKey));
|
||||
await driver.InitializeAsync(ConfigJson, CancellationToken.None);
|
||||
var before = driver.GetHealth().LastSuccessfulRead.ShouldNotBeNull();
|
||||
await Task.Delay(20, TestContext.Current.CancellationToken);
|
||||
|
||||
driver.ObservePollOutcome([new DataValueSnapshot(null, statusCode, DateTime.UtcNow, DateTime.UtcNow)]);
|
||||
|
||||
driver.GetHealth().LastSuccessfulRead.ShouldNotBeNull().ShouldBeGreaterThan(before);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(SqlStatusCodes.BadTimeout)] // the database did not answer at all
|
||||
[InlineData(SqlStatusCodes.BadNodeIdUnknown)] // the resolver never reached the database
|
||||
[InlineData(SqlStatusCodes.BadConfigurationError)] // the planner rejected the tag; nothing was executed
|
||||
[InlineData(SqlStatusCodes.BadInternalError)] // a reader defect; no statement about the database
|
||||
public async Task ObservePollOutcome_aCodeThatIsNoEvidenceOfAnAnswer_leavesTheEvidenceClockAlone(
|
||||
uint statusCode)
|
||||
{
|
||||
using var fixture = new SqlitePollFixture();
|
||||
await using var driver = NewDriver(fixture, KvEntry("Speed", SqlitePollFixture.PresentKey));
|
||||
await driver.InitializeAsync(ConfigJson, CancellationToken.None);
|
||||
var before = driver.GetHealth().LastSuccessfulRead.ShouldNotBeNull();
|
||||
await Task.Delay(20, TestContext.Current.CancellationToken);
|
||||
|
||||
driver.ObservePollOutcome([new DataValueSnapshot(null, statusCode, DateTime.UtcNow, DateTime.UtcNow)]);
|
||||
|
||||
driver.GetHealth().LastSuccessfulRead.ShouldBe(before);
|
||||
}
|
||||
|
||||
// ---- Driver.Sql-010: the health field is read-modify-written from several poll loops ----
|
||||
|
||||
[Fact]
|
||||
public async Task HealthTransitions_fromConcurrentPolls_neverRewindTheEvidenceClock()
|
||||
{
|
||||
// Driver.Sql-010. Every health transition is a read-modify-write of one field: Degrade reads the
|
||||
// current snapshot to PRESERVE LastSuccessfulRead, then writes a new record. With a plain
|
||||
// Volatile read + write, a Degrade that read the clock at t0 can be published after a concurrent
|
||||
// advance to t1 — so the clock goes BACKWARDS, and the advance is lost. That is not cosmetic: the
|
||||
// comms-loss watchdog measures how long the clock has been frozen, so a lost advance is a spurious
|
||||
// 90 s reconnect. Monotonicity is the observable the lost update violates, and it is exactly what a
|
||||
// compare-and-swap guarantees.
|
||||
//
|
||||
// A driver with several subscriptions runs one poll loop each, plus any ad-hoc ReadAsync — so two
|
||||
// classifications landing at once is the ordinary case, not a contrived one.
|
||||
using var fixture = new SqlitePollFixture();
|
||||
await using var driver = NewDriver(fixture, KvEntry("Speed", SqlitePollFixture.PresentKey));
|
||||
await driver.InitializeAsync(ConfigJson, CancellationToken.None);
|
||||
|
||||
const int iterations = 30_000;
|
||||
var advancing = new DataValueSnapshot(1.0, SqlStatusCodes.Good, DateTime.UtcNow, DateTime.UtcNow);
|
||||
var degrading = new DataValueSnapshot(null, SqlStatusCodes.BadTimeout, DateTime.UtcNow, DateTime.UtcNow);
|
||||
using var done = new CancellationTokenSource();
|
||||
|
||||
var rewinds = 0;
|
||||
var token = TestContext.Current.CancellationToken;
|
||||
var observer = Task.Run(
|
||||
() =>
|
||||
{
|
||||
var highWater = driver.GetHealth().LastSuccessfulRead ?? DateTime.MinValue;
|
||||
while (!done.IsCancellationRequested)
|
||||
{
|
||||
var seen = driver.GetHealth().LastSuccessfulRead ?? DateTime.MinValue;
|
||||
if (seen < highWater) Interlocked.Increment(ref rewinds);
|
||||
else highWater = seen;
|
||||
}
|
||||
},
|
||||
token);
|
||||
|
||||
var writers = new[]
|
||||
{
|
||||
Task.Run(
|
||||
() => { for (var i = 0; i < iterations; i++) driver.ObservePollOutcome([advancing]); }, token),
|
||||
Task.Run(
|
||||
() => { for (var i = 0; i < iterations; i++) driver.ObservePollOutcome([degrading]); }, token),
|
||||
};
|
||||
|
||||
await Task.WhenAll(writers);
|
||||
await done.CancelAsync();
|
||||
await observer;
|
||||
|
||||
Volatile.Read(ref rewinds).ShouldBe(0);
|
||||
}
|
||||
|
||||
// ---- ISubscribable (poll-engine overlay) ----
|
||||
|
||||
[Fact]
|
||||
@@ -391,23 +559,92 @@ public sealed class SqlDriverTests
|
||||
{
|
||||
// On a subscribed-read outage the reader throws a DbException; ReadAsync classifies health with a
|
||||
// good, endpoint-bearing message + host Stopped and rethrows the SAME exception, which the poll
|
||||
// engine then routes to HandlePollError. HandlePollError must ignore the DbException class ReadAsync
|
||||
// owns — pre-fix it re-degraded with the bare ex.Message (the WORSE of the two LastErrors, and a
|
||||
// credential-leak path) and logged the outage a second time.
|
||||
// engine then routes to HandlePollError. HandlePollError must leave the classification ReadAsync
|
||||
// already made alone — pre-fix it re-degraded with the bare ex.Message (the WORSE of the two
|
||||
// LastErrors, and a credential-leak path) and logged the outage a second time.
|
||||
// Driven through the REAL round trip rather than by handing HandlePollError a bare exception: the
|
||||
// deferral is owned by ReadAsync marking the throw, not by the exception's type (Driver.Sql-004).
|
||||
using var fixture = new SqlitePollFixture();
|
||||
var logger = new CapturingLogger();
|
||||
await using var driver = NewDriver(fixture, logger, KvEntry("Speed", SqlitePollFixture.PresentKey));
|
||||
var factory = new BreakableFactory(fixture.Factory);
|
||||
await using var driver = NewDriver(
|
||||
fixture, fixture.ConnectionString, logger, factory, KvEntry("Speed", SqlitePollFixture.PresentKey));
|
||||
await driver.InitializeAsync(ConfigJson, CancellationToken.None);
|
||||
driver.GetHealth().State.ShouldBe(DriverState.Healthy);
|
||||
|
||||
factory.BreakByThrowingOnOpen("connection failed; Password=" + SecretToken);
|
||||
var thrown = await Should.ThrowAsync<DbException>(
|
||||
async () => await driver.ReadAsync(["Speed"], CancellationToken.None));
|
||||
|
||||
var classified = driver.GetHealth();
|
||||
classified.State.ShouldBe(DriverState.Degraded);
|
||||
classified.LastError.ShouldNotBeNull();
|
||||
classified.LastError.ShouldContain("could not reach");
|
||||
var warningsAfterRead = logger.Entries.Count(e => e.Level == LogLevel.Warning);
|
||||
|
||||
driver.HandlePollError(thrown);
|
||||
|
||||
var health = driver.GetHealth();
|
||||
// Untouched: HandlePollError left the exception to ReadAsync, so it neither re-degraded nor leaked…
|
||||
health.LastError.ShouldNotBeNull().ShouldBe(classified.LastError);
|
||||
health.LastError.ShouldNotContain(SecretToken);
|
||||
// …and it logged nothing (ReadAsync owns the single outage warning).
|
||||
logger.Entries.Count(e => e.Level == LogLevel.Warning).ShouldBe(warningsAfterRead);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task HandlePollError_forANonDbExceptionReadAsyncClassified_stillDefers_soTheGoodLastErrorSurvives()
|
||||
{
|
||||
// Driver.Sql-004. Not every unreachable-database throw is a DbException: the reader raises a plain
|
||||
// InvalidOperationException when the provider factory hands back no connection, and a provider can
|
||||
// raise ArgumentException/FormatException off a malformed connection string on the same open path.
|
||||
// ReadAsync classifies those exactly as it classifies a DbException — endpoint-bearing LastError,
|
||||
// host Stopped — and rethrows, so HandlePollError must defer to it here too. RED against the
|
||||
// `ex is DbException` type test, which re-degraded with the guidance-free "poll failed: TypeName"
|
||||
// and logged the outage twice.
|
||||
using var fixture = new SqlitePollFixture();
|
||||
var logger = new CapturingLogger();
|
||||
var factory = new BreakableFactory(fixture.Factory);
|
||||
await using var driver = NewDriver(
|
||||
fixture, fixture.ConnectionString, logger, factory, KvEntry("Speed", SqlitePollFixture.PresentKey));
|
||||
await driver.InitializeAsync(ConfigJson, CancellationToken.None);
|
||||
|
||||
factory.BreakByReturningNoConnection();
|
||||
var thrown = await Should.ThrowAsync<InvalidOperationException>(
|
||||
async () => await driver.ReadAsync(["Speed"], CancellationToken.None));
|
||||
|
||||
var classified = driver.GetHealth();
|
||||
classified.State.ShouldBe(DriverState.Degraded);
|
||||
classified.LastError.ShouldNotBeNull();
|
||||
classified.LastError.ShouldContain("could not reach");
|
||||
driver.GetHostStatuses()[0].State.ShouldBe(HostState.Stopped);
|
||||
var warningsAfterRead = logger.Entries.Count(e => e.Level == LogLevel.Warning);
|
||||
|
||||
driver.HandlePollError(thrown);
|
||||
|
||||
var health = driver.GetHealth();
|
||||
health.LastError.ShouldNotBeNull()
|
||||
.ShouldBe(classified.LastError); // the richer message survived…
|
||||
health.LastError.ShouldNotContain("poll failed"); // …and was not overwritten by the poor one
|
||||
logger.Entries.Count(e => e.Level == LogLevel.Warning).ShouldBe(warningsAfterRead);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task HandlePollError_forAnExceptionReadAsyncNeverSaw_degrades_evenWhenItIsADbException()
|
||||
{
|
||||
// The falsifiability control for the two deferrals above: the marker is per-exception, not per-type,
|
||||
// so a DbException that never went through ReadAsync's classification is NOT owned by it and must
|
||||
// still reach the health surface. A type test would silently swallow this one.
|
||||
using var fixture = new SqlitePollFixture();
|
||||
var driver = NewDriver(fixture, KvEntry("Speed", SqlitePollFixture.PresentKey));
|
||||
|
||||
driver.HandlePollError(new FakeDbException("connection failed; Password=" + SecretToken));
|
||||
|
||||
var health = driver.GetHealth();
|
||||
// Untouched: HandlePollError left the DbException to ReadAsync, so it neither degraded nor leaked…
|
||||
health.State.ShouldBe(DriverState.Healthy);
|
||||
health.LastError.ShouldBeNull();
|
||||
// …and it logged nothing (ReadAsync owns the single outage warning).
|
||||
logger.Entries.Count(e => e.Level == LogLevel.Warning).ShouldBe(0);
|
||||
health.State.ShouldBe(DriverState.Degraded);
|
||||
health.LastError.ShouldNotBeNull();
|
||||
health.LastError.ShouldNotContain(SecretToken); // type only, never the message text
|
||||
health.LastError.ShouldContain(nameof(FakeDbException));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -518,6 +755,26 @@ public sealed class SqlDriverTests
|
||||
operationTimeout: TimeSpan.FromSeconds(15), commandTimeout: TimeSpan.FromSeconds(10),
|
||||
rawTags: rawTags);
|
||||
|
||||
/// <summary>The same driver, over a caller-supplied provider factory — the outage-injection seam.</summary>
|
||||
private static SqlDriver NewDriver(
|
||||
SqlitePollFixture fixture,
|
||||
string connectionString,
|
||||
CapturingLogger logger,
|
||||
DbProviderFactory factory,
|
||||
params RawTagEntry[] rawTags)
|
||||
=> new(
|
||||
new SqlDriverOptions
|
||||
{
|
||||
RawTags = rawTags,
|
||||
OperationTimeout = TimeSpan.FromSeconds(15),
|
||||
CommandTimeout = TimeSpan.FromSeconds(10),
|
||||
},
|
||||
DriverInstanceId,
|
||||
new SqliteDialect(),
|
||||
connectionString,
|
||||
factory: factory,
|
||||
logger: logger);
|
||||
|
||||
/// <summary>
|
||||
/// Builds the driver through its production constructor. That constructor <b>is</b> the injection
|
||||
/// seam — dialect, provider factory and already-resolved connection string are all parameters — so no
|
||||
@@ -668,6 +925,28 @@ public sealed class SqlDriverTests
|
||||
public override DbConnection CreateConnection() => new ThrowingConnection(openFailureMessage);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A provider factory that behaves exactly like the fixture's until it is broken, then reproduces one
|
||||
/// of the two open-path outages the driver has to classify: a factory that returns <b>no</b>
|
||||
/// connection (a plain <see cref="InvalidOperationException"/> from the reader — NOT a
|
||||
/// <see cref="DbException"/>), or one whose connections throw on open.
|
||||
/// </summary>
|
||||
private sealed class BreakableFactory(DbProviderFactory inner) : DbProviderFactory
|
||||
{
|
||||
private Func<DbConnection?>? _broken;
|
||||
|
||||
/// <summary>Hands back no connection, the shape that makes the reader raise InvalidOperationException.</summary>
|
||||
public void BreakByReturningNoConnection() => Volatile.Write(ref _broken, static () => null);
|
||||
|
||||
/// <summary>Hands back a connection that fails to open with <paramref name="message"/>.</summary>
|
||||
/// <param name="message">The provider message, which may carry credentials.</param>
|
||||
public void BreakByThrowingOnOpen(string message)
|
||||
=> Volatile.Write(ref _broken, () => new ThrowingConnection(message));
|
||||
|
||||
public override DbConnection? CreateConnection()
|
||||
=> Volatile.Read(ref _broken) is { } broken ? broken() : inner.CreateConnection();
|
||||
}
|
||||
|
||||
/// <summary>A connection that throws <see cref="FakeDbException"/> on open; every other member is inert.</summary>
|
||||
private sealed class ThrowingConnection(string openFailureMessage) : DbConnection
|
||||
{
|
||||
|
||||
Reference in New Issue
Block a user