Joseph Doherty
e3e3f5fb04
Three review findings against SqlPollReader (967e5140).
1. WideRow's where-pair form emits no ORDER BY and no row limit, so a
selector that is not actually unique made the reader publish an
arbitrary, storage-order-dependent row — silently, unlike IndexByKey's
duplicate-key warning. Row selection moves out of MapMember into a new
SelectWideRow, which is called once per group and emits the same
rate-limited contract warning (naming table, selector and row count).
The row-limit half of the suggested fix was deliberately NOT taken: a
TOP 1 with no ORDER BY is not deterministic either, and it would
destroy the Rows.Count > 1 evidence the warning depends on, turning a
loud misconfiguration into a permanently silent one. Rationale is in
SelectWideRow's docs. No SQL changed, so no planner golden string moved.
2. The "a frozen database can never accumulate more than
maxConcurrentGroups connections" claim was asserted in the docs and
untested. ReadAsync_aTimedOutGroupKeepsItsConcurrencySlotUntilTheQuery-
TrulyFinishes wedges a group behind an EXCLUSIVE transaction, times it
out, and proves the slot is still held (a second poll on a cap of 1
times out on the gate without ever reaching the factory) and is handed
back only once the abandoned query truly finishes. Moving the release
from the work to the waiter fails this test and no other.
3. Task.Run thread-pool starvation: documented, not changed.
Microsoft.Data.SqlClient's async path parks no pool thread on a frozen
server, so the driver cannot starve itself; LongRunning would cost a
dedicated OS thread per group per poll forever to insure against a
provider this driver does not ship. RunGroupAsync now records the
decision and what a BadTimeout does and does not mean, for the
blackhole gate.
Also: the timeout warning now names the group's table, so a BadTimeout
storm identifies which source is frozen.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
840 lines
44 KiB
C#
840 lines
44 KiB
C#
using System.Data.Common;
|
||
using System.Diagnostics;
|
||
using System.Globalization;
|
||
using Microsoft.Extensions.Logging;
|
||
using Microsoft.Extensions.Logging.Abstractions;
|
||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||
using ZB.MOM.WW.OtOpcUa.Driver.Sql.Contracts;
|
||
|
||
namespace ZB.MOM.WW.OtOpcUa.Driver.Sql;
|
||
|
||
/// <summary>
|
||
/// Executes one poll pass: resolves each requested reference to a <see cref="SqlTagDefinition"/>, folds
|
||
/// the definitions into the minimum number of queries via <see cref="SqlGroupPlanner"/>, runs one
|
||
/// command per group under a bounded deadline, and slices each result set back to per-tag
|
||
/// <see cref="DataValueSnapshot"/>s — <b>positionally aligned with the caller's reference list</b>
|
||
/// (design §3.2). Structurally the SQL analogue of <c>ModbusDriver.ReadCoalescedAsync</c>: group → one
|
||
/// round-trip → slice back.
|
||
/// <para><b>N in, N out, in order.</b> The returned list always has exactly one entry per requested
|
||
/// reference, at the same index, whatever happened. Per-tag failures are Bad-coded snapshots, never
|
||
/// exceptions (the <see cref="IReadable"/> contract) — see <see cref="SqlStatusCodes"/> for which code
|
||
/// means what. This matters more here than in a point-to-point driver: one result set feeds many tags,
|
||
/// so a slicing defect does not fail loudly — it publishes <em>one tag's value onto another tag's
|
||
/// node</em>.</para>
|
||
/// <para><b>The whole call throws only when the database is unreachable</b> — that is, when opening a
|
||
/// connection fails. That surfaces to <see cref="PollGroupEngine"/> as a poll failure and earns the
|
||
/// capped-exponential backoff; the driver's <c>IHostConnectivityProbe</c> is what scopes Bad quality
|
||
/// across the subtree during an outage, so nothing is lost by not manufacturing per-tag snapshots for
|
||
/// an outage. A query that fails <em>after</em> the connection opened is the group's problem, not the
|
||
/// server's, and Bad-codes that group only.</para>
|
||
/// <para><b>Deadlines (design §8.3, the R2-01 frozen-peer lesson).</b> Two independent mechanisms, both
|
||
/// required. <c>CommandTimeout</c> is the <em>server-side</em> backstop and bounds nothing on a wedged
|
||
/// socket. The wall-clock bound is client-side and is what actually guarantees this method returns: see
|
||
/// <see cref="RunGroupAsync"/>. A frozen database yields <see cref="SqlStatusCodes.BadTimeout"/>
|
||
/// snapshots — it must never wedge the poll thread.</para>
|
||
/// <para><b>Not a driver.</b> This type owns the read path only; it deliberately holds no health state,
|
||
/// no subscription state and no connection between polls, so the driver shell can wrap it in
|
||
/// <see cref="PollGroupEngine"/> and own those concerns.</para>
|
||
/// </summary>
|
||
public sealed class SqlPollReader
|
||
{
|
||
/// <summary>Minimum spacing between "your source violates the query contract" warnings, per reader.</summary>
|
||
private static readonly TimeSpan ContractWarningInterval = TimeSpan.FromMinutes(1);
|
||
|
||
private readonly DbProviderFactory _factory;
|
||
private readonly string _connectionString;
|
||
private readonly ISqlDialect _dialect;
|
||
private readonly TimeSpan _operationTimeout;
|
||
private readonly int _commandTimeoutSeconds;
|
||
private readonly bool _nullIsBad;
|
||
private readonly Func<string, SqlTagDefinition?> _resolve;
|
||
private readonly ILogger _logger;
|
||
|
||
/// <summary>
|
||
/// Caps concurrently executing group queries — and therefore concurrently held connections
|
||
/// (design §8.4). Never disposed: a <see cref="SemaphoreSlim"/> only needs disposal once its
|
||
/// <c>AvailableWaitHandle</c> has been touched, and not disposing it also removes the hazard of a
|
||
/// timed-out-but-still-running group releasing into a disposed semaphore.
|
||
/// </summary>
|
||
private readonly SemaphoreSlim _gate;
|
||
|
||
private long _lastContractWarningTicks;
|
||
|
||
/// <summary>Initializes a new instance of the <see cref="SqlPollReader"/> class.</summary>
|
||
/// <param name="factory">
|
||
/// Creates the provider's connections. Passed explicitly rather than taken from
|
||
/// <paramref name="dialect"/> so the driver can hand in an instrumented or pre-configured factory.
|
||
/// </param>
|
||
/// <param name="connectionString">
|
||
/// The resolved connection string (never the authored <c>connectionStringRef</c> — secrets are
|
||
/// resolved by the driver at Initialize, design §8.2).
|
||
/// </param>
|
||
/// <param name="dialect">Supplies identifier quoting, row-limit syntax, and column-type mapping.</param>
|
||
/// <param name="commandTimeout">
|
||
/// The ADO.NET <c>CommandTimeout</c> backstop. Rounded <b>up</b> to whole seconds and floored at 1,
|
||
/// because ADO.NET reads <c>CommandTimeout = 0</c> as "wait forever" — the one value this option
|
||
/// must never silently become.
|
||
/// </param>
|
||
/// <param name="operationTimeout">
|
||
/// The wall-clock ceiling on one group's whole operation — waiting for a concurrency slot, opening
|
||
/// the connection, and running the query. A breach Bad-codes that group with
|
||
/// <see cref="SqlStatusCodes.BadTimeout"/>.
|
||
/// <para>Authoring should keep this strictly greater than <paramref name="commandTimeout"/>
|
||
/// (design §8.3): inverted, the client-side abort always fires first and masks the server-side
|
||
/// backstop. That rule is <b>not</b> enforced here — it belongs to config validation, and this type
|
||
/// stays usable for the tests that must deliberately invert it.</para>
|
||
/// </param>
|
||
/// <param name="maxConcurrentGroups">
|
||
/// Maximum group queries — and connections — in flight at once. A timed-out group keeps its slot
|
||
/// until it truly finishes, so this is a hard ceiling on connections even against a frozen database.
|
||
/// See <see cref="RunGroupAsync"/> for why a high value is safe under
|
||
/// <c>Microsoft.Data.SqlClient</c> but wants sizing against thread-pool headroom under a provider
|
||
/// that blocks synchronously.
|
||
/// </param>
|
||
/// <param name="nullIsBad">
|
||
/// <see langword="true"/> publishes <see cref="SqlStatusCodes.Bad"/> for a NULL value cell instead
|
||
/// of the default <see cref="SqlStatusCodes.Uncertain"/>. It governs a <em>present</em> row with a
|
||
/// NULL cell only — an absent row is always <see cref="SqlStatusCodes.BadNoData"/>.
|
||
/// </param>
|
||
/// <param name="resolve">
|
||
/// RawPath → tag definition; <see langword="null"/> on a miss. Shaped to match
|
||
/// <see cref="EquipmentTagRefResolver{TDef}"/>'s lookup so the driver can pass its authored table
|
||
/// straight through.
|
||
/// </param>
|
||
/// <param name="logger">Optional; defaults to a no-op logger.</param>
|
||
/// <exception cref="ArgumentNullException">A required reference argument is null.</exception>
|
||
/// <exception cref="ArgumentException"><paramref name="connectionString"/> is blank.</exception>
|
||
/// <exception cref="ArgumentOutOfRangeException">A timeout is non-positive, or the cap is below 1.</exception>
|
||
public SqlPollReader(
|
||
DbProviderFactory factory,
|
||
string connectionString,
|
||
ISqlDialect dialect,
|
||
TimeSpan commandTimeout,
|
||
TimeSpan operationTimeout,
|
||
int maxConcurrentGroups,
|
||
bool nullIsBad,
|
||
Func<string, SqlTagDefinition?> resolve,
|
||
ILogger? logger = null)
|
||
{
|
||
ArgumentNullException.ThrowIfNull(factory);
|
||
ArgumentNullException.ThrowIfNull(dialect);
|
||
ArgumentNullException.ThrowIfNull(resolve);
|
||
if (string.IsNullOrWhiteSpace(connectionString))
|
||
throw new ArgumentException("A Sql poll reader needs a connection string.", nameof(connectionString));
|
||
ArgumentOutOfRangeException.ThrowIfLessThanOrEqual(commandTimeout, TimeSpan.Zero);
|
||
ArgumentOutOfRangeException.ThrowIfLessThanOrEqual(operationTimeout, TimeSpan.Zero);
|
||
ArgumentOutOfRangeException.ThrowIfLessThan(maxConcurrentGroups, 1);
|
||
|
||
_factory = factory;
|
||
_connectionString = connectionString;
|
||
_dialect = dialect;
|
||
_operationTimeout = operationTimeout;
|
||
_commandTimeoutSeconds = Math.Max(1, (int)Math.Ceiling(commandTimeout.TotalSeconds));
|
||
_nullIsBad = nullIsBad;
|
||
_resolve = resolve;
|
||
_logger = logger ?? NullLogger.Instance;
|
||
_gate = new SemaphoreSlim(maxConcurrentGroups, maxConcurrentGroups);
|
||
}
|
||
|
||
/// <summary>
|
||
/// Reads every reference in one poll pass. Matches <see cref="IReadable.ReadAsync"/>'s shape so the
|
||
/// driver shell can delegate to it directly.
|
||
/// </summary>
|
||
/// <param name="fullReferences">The RawPaths to read. Empty yields an empty result and touches no connection.</param>
|
||
/// <param name="cancellationToken">
|
||
/// The caller's token. Its cancellation <b>propagates</b> as an
|
||
/// <see cref="OperationCanceledException"/> — the engine asked the poll to stop, and nobody is
|
||
/// waiting for the snapshots. This is deliberately asymmetric with an <c>operationTimeout</c>
|
||
/// breach, which is a data-quality fact clients must see and so becomes
|
||
/// <see cref="SqlStatusCodes.BadTimeout"/> snapshots.
|
||
/// </param>
|
||
/// <returns>One snapshot per reference, at the same index.</returns>
|
||
/// <exception cref="ArgumentNullException"><paramref name="fullReferences"/> is null.</exception>
|
||
/// <exception cref="OperationCanceledException"><paramref name="cancellationToken"/> was cancelled.</exception>
|
||
/// <exception cref="DbException">The database could not be reached (opening a connection failed).</exception>
|
||
public async Task<IReadOnlyList<DataValueSnapshot>> ReadAsync(
|
||
IReadOnlyList<string> fullReferences, CancellationToken cancellationToken)
|
||
{
|
||
ArgumentNullException.ThrowIfNull(fullReferences);
|
||
|
||
var results = new DataValueSnapshot[fullReferences.Count];
|
||
if (results.Length == 0) return results;
|
||
|
||
var readAt = DateTime.UtcNow;
|
||
|
||
// Resolve first. A miss is this tag's own problem (BadNodeIdUnknown) and must not reach the planner,
|
||
// whose members are the sole thing the slice-back walks.
|
||
var definitions = new List<SqlTagDefinition>(fullReferences.Count);
|
||
var slots = new Dictionary<SqlTagDefinition, List<int>>();
|
||
for (var i = 0; i < fullReferences.Count; i++)
|
||
{
|
||
var definition = fullReferences[i] is { } reference ? _resolve(reference) : null;
|
||
if (definition is null)
|
||
{
|
||
results[i] = new DataValueSnapshot(null, SqlStatusCodes.BadNodeIdUnknown, null, readAt);
|
||
continue;
|
||
}
|
||
|
||
definitions.Add(definition);
|
||
if (!slots.TryGetValue(definition, out var positions))
|
||
slots[definition] = positions = [];
|
||
positions.Add(i);
|
||
}
|
||
|
||
if (definitions.Count > 0)
|
||
await ReadGroupsAsync(definitions, slots, results, readAt, cancellationToken).ConfigureAwait(false);
|
||
|
||
// Fail-safe. Nothing above should leave a hole, but "N in, N out" is a contract the caller indexes
|
||
// blind — a null here would be a NullReferenceException in the publish fan-out, far from its cause.
|
||
for (var i = 0; i < results.Length; i++)
|
||
results[i] ??= new DataValueSnapshot(null, SqlStatusCodes.BadInternalError, null, readAt);
|
||
|
||
return results;
|
||
}
|
||
|
||
/// <summary>Plans the resolved definitions, runs every group concurrently, and slices the results back.</summary>
|
||
private async Task ReadGroupsAsync(
|
||
List<SqlTagDefinition> definitions,
|
||
Dictionary<SqlTagDefinition, List<int>> slots,
|
||
DataValueSnapshot[] results,
|
||
DateTime readAt,
|
||
CancellationToken cancellationToken)
|
||
{
|
||
IReadOnlyList<SqlQueryPlan> plans;
|
||
try
|
||
{
|
||
plans = SqlGroupPlanner.Plan(definitions, _dialect);
|
||
}
|
||
catch (Exception ex) when (ex is ArgumentException or NotSupportedException)
|
||
{
|
||
// A definition with a broken invariant — SqlEquipmentTagParser rejects all of these up front, so
|
||
// reaching here means a definition was built past the parser. It is an authoring fault, not a
|
||
// database fault: Bad-code the planned tags rather than throwing and triggering connection backoff.
|
||
_logger.LogError(ex, "Sql poll planning failed for {Count} tag(s); publishing BadConfigurationError.",
|
||
definitions.Count);
|
||
foreach (var definition in definitions)
|
||
{
|
||
if (!slots.TryGetValue(definition, out var positions)) continue;
|
||
var snapshot = new DataValueSnapshot(null, SqlStatusCodes.BadConfigurationError, null, readAt);
|
||
foreach (var position in positions) results[position] = snapshot;
|
||
}
|
||
|
||
return;
|
||
}
|
||
|
||
// Groups run concurrently; _gate — not the task count — is what bounds open connections. Each group
|
||
// carries its own deadline, so the whole call is bounded by ~operationTimeout rather than by
|
||
// plans.Count × operationTimeout.
|
||
var groups = new Task<GroupResult>[plans.Count];
|
||
for (var g = 0; g < plans.Count; g++)
|
||
groups[g] = RunGroupAsync(plans[g], cancellationToken);
|
||
|
||
// WhenAll waits for every group before surfacing a fault, so a thrown "database unreachable" never
|
||
// leaves a sibling group running against a connection nobody is watching.
|
||
var outcomes = await Task.WhenAll(groups).ConfigureAwait(false);
|
||
|
||
for (var g = 0; g < plans.Count; g++)
|
||
Slice(plans[g], outcomes[g], slots, results, readAt);
|
||
}
|
||
|
||
/// <summary>
|
||
/// Runs one group's query under a hard wall-clock ceiling.
|
||
/// <para><b>Why the ceiling is <c>Task.WaitAsync</c> and not just a linked
|
||
/// <c>CancelAfter</c>.</b> A linked token only bounds a
|
||
/// provider that <em>honours</em> it. The S7 R2-01 finding was exactly a provider whose async path
|
||
/// ignored its deadline, and ADO.NET has the same shape: some providers implement
|
||
/// <c>ExecuteReaderAsync</c> synchronously, and even a genuinely async one can hang inside its own
|
||
/// cancellation handshake against a frozen socket. Both are used here: the linked token so a
|
||
/// well-behaved provider unwinds cleanly and releases its connection, and <c>WaitAsync</c> so
|
||
/// control returns at the deadline regardless.</para>
|
||
/// <para>The work is started on the thread pool so a provider that blocks synchronously blocks a
|
||
/// pool thread rather than the caller — without that, a synchronous provider never yields the task
|
||
/// there would be to bound.</para>
|
||
/// <para><b>Thread-pool scheduling and what a BadTimeout therefore means.</b> The group's clock
|
||
/// starts before <c>Task.Run</c>, so in principle a group could burn budget queueing for a pool
|
||
/// thread rather than waiting on the database. <c>TaskCreationOptions.LongRunning</c> (a dedicated
|
||
/// OS thread) was considered and <b>rejected</b>: it would create and tear down one thread per group
|
||
/// per poll, forever, on a driver whose whole job is to poll on a short cadence — a permanent cost
|
||
/// paid against a hazard that cannot arise with the provider this driver actually ships.
|
||
/// <c>Microsoft.Data.SqlClient</c>'s async path is genuinely asynchronous: the delegate reaches its
|
||
/// first real await within microseconds and returns the pool thread, so a frozen SQL Server parks
|
||
/// <em>zero</em> pool threads however long it stays frozen, and the driver cannot starve itself no
|
||
/// matter how high <c>maxConcurrentGroups</c> goes. The queueing scenario needs a provider that
|
||
/// degrades to synchronous blocking — which is exactly what the SQLite test rig exploits on purpose,
|
||
/// and is not a shipped configuration. <b>Consequence for outage diagnosis (e.g. blackhole-testing a
|
||
/// paused SQL Server):</b> under <c>Microsoft.Data.SqlClient</c> a
|
||
/// <see cref="SqlStatusCodes.BadTimeout"/> means the database did not answer inside
|
||
/// <c>operationTimeout</c> — it cannot be an artefact of this driver's own pool pressure. Under a
|
||
/// synchronously-blocking provider it may also include queueing delay, so size
|
||
/// <c>maxConcurrentGroups</c> below the process's readily-available worker count there.</para>
|
||
/// <para><b>The concurrency slot is released by the work, not by the waiter.</b> A timed-out group
|
||
/// keeps its slot until it truly finishes, so a frozen database can never accumulate more than
|
||
/// <c>maxConcurrentGroups</c> connections however long it stays frozen. The slot wait is itself
|
||
/// inside the deadline, so a poll behind a wedged group still returns on time — as BadTimeout.</para>
|
||
/// </summary>
|
||
private async Task<GroupResult> RunGroupAsync(SqlQueryPlan plan, CancellationToken cancellationToken)
|
||
{
|
||
var clock = Stopwatch.StartNew();
|
||
|
||
if (!await _gate.WaitAsync(Remaining(clock), cancellationToken).ConfigureAwait(false))
|
||
return GroupResult.TimedOut;
|
||
|
||
var handedOff = false;
|
||
var deadline = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
|
||
Task<GroupResult> work;
|
||
try
|
||
{
|
||
var budget = Remaining(clock);
|
||
if (budget <= TimeSpan.Zero) return GroupResult.TimedOut;
|
||
deadline.CancelAfter(budget);
|
||
|
||
work = Task.Run(
|
||
async () =>
|
||
{
|
||
try
|
||
{
|
||
return await QueryAsync(plan, deadline.Token).ConfigureAwait(false);
|
||
}
|
||
finally
|
||
{
|
||
_gate.Release();
|
||
deadline.Dispose();
|
||
}
|
||
},
|
||
CancellationToken.None);
|
||
handedOff = true;
|
||
}
|
||
finally
|
||
{
|
||
// Ownership of both the slot and the CTS transfers to the work task; release them here only on
|
||
// the paths where that task was never created.
|
||
if (!handedOff)
|
||
{
|
||
_gate.Release();
|
||
deadline.Dispose();
|
||
}
|
||
}
|
||
|
||
try
|
||
{
|
||
return await work.WaitAsync(Remaining(clock), cancellationToken).ConfigureAwait(false);
|
||
}
|
||
catch (TimeoutException)
|
||
{
|
||
// Our wall-clock bound fired while the provider was still inside the call.
|
||
return TimedOut(plan, work);
|
||
}
|
||
catch (OperationCanceledException)
|
||
{
|
||
// Either the caller pulled the plug, or our deadline fired and the provider DID honour the
|
||
// linked token. Both leave the work running with nobody to observe its fault.
|
||
if (!cancellationToken.IsCancellationRequested) return TimedOut(plan, work);
|
||
Detach(work);
|
||
throw;
|
||
}
|
||
}
|
||
|
||
/// <summary>How much of this group's <c>operationTimeout</c> budget is left; never negative.</summary>
|
||
private TimeSpan Remaining(Stopwatch clock)
|
||
{
|
||
var remaining = _operationTimeout - clock.Elapsed;
|
||
return remaining > TimeSpan.Zero ? remaining : TimeSpan.Zero;
|
||
}
|
||
|
||
/// <summary>Records the deadline breach and detaches the abandoned work.</summary>
|
||
private GroupResult TimedOut(SqlQueryPlan plan, Task work)
|
||
{
|
||
// The table is named for the same reason the contract warnings name it: under a partial outage this
|
||
// line repeats every poll, and "which source is frozen" is the only question the operator has.
|
||
_logger.LogWarning(
|
||
"Sql poll group ({Model}, '{Table}', {Members} tag(s)) exceeded the {Timeout} ms operation " +
|
||
"timeout; publishing BadTimeout.",
|
||
plan.Model, plan.Members[0].Table, plan.Members.Count,
|
||
(int)_operationTimeout.TotalMilliseconds);
|
||
|
||
Detach(work);
|
||
return GroupResult.TimedOut;
|
||
}
|
||
|
||
/// <summary>
|
||
/// Observes an abandoned group's eventual fault so it never surfaces as an unobserved task
|
||
/// exception. The task still owns its connection and still releases the concurrency slot when it
|
||
/// finally completes — that is what keeps a frozen database from accumulating connections.
|
||
/// </summary>
|
||
private static void Detach(Task work)
|
||
=> _ = work.ContinueWith(
|
||
static t => _ = t.Exception,
|
||
CancellationToken.None,
|
||
TaskContinuationOptions.OnlyOnFaulted | TaskContinuationOptions.ExecuteSynchronously,
|
||
TaskScheduler.Default);
|
||
|
||
/// <summary>
|
||
/// Opens a connection, executes the plan's single command, and materialises the result set.
|
||
/// <para>The rows are read into memory <b>before</b> the reader is disposed — the slice-back needs
|
||
/// random access across members, and holding a reader (and therefore a connection) open across that
|
||
/// work is exactly how a poll loop exhausts a pool.</para>
|
||
/// </summary>
|
||
private async Task<GroupResult> QueryAsync(SqlQueryPlan plan, CancellationToken cancellationToken)
|
||
{
|
||
var connection = _factory.CreateConnection()
|
||
?? throw new InvalidOperationException(
|
||
$"The {_dialect.Provider} provider factory returned no connection.");
|
||
|
||
await using (connection.ConfigureAwait(false))
|
||
{
|
||
connection.ConnectionString = _connectionString;
|
||
|
||
// Deliberately OUTSIDE the catch below: a failed open means the database is unreachable, which
|
||
// is the one condition IReadable says the whole call may throw on (→ PollGroupEngine backoff).
|
||
await connection.OpenAsync(cancellationToken).ConfigureAwait(false);
|
||
|
||
try
|
||
{
|
||
var command = connection.CreateCommand();
|
||
await using (command.ConfigureAwait(false))
|
||
{
|
||
command.CommandText = plan.SqlText;
|
||
command.CommandTimeout = _commandTimeoutSeconds;
|
||
for (var i = 0; i < plan.ParameterNames.Count; i++)
|
||
{
|
||
var parameter = command.CreateParameter();
|
||
// Bound by the plan's own index pairing — never by re-deriving the marker convention.
|
||
parameter.ParameterName = plan.ParameterNames[i];
|
||
parameter.Value = plan.Parameters[i] ?? DBNull.Value;
|
||
command.Parameters.Add(parameter);
|
||
}
|
||
|
||
var reader = await command.ExecuteReaderAsync(cancellationToken).ConfigureAwait(false);
|
||
await using (reader.ConfigureAwait(false))
|
||
{
|
||
return await MaterialiseAsync(plan, reader, cancellationToken).ConfigureAwait(false);
|
||
}
|
||
}
|
||
}
|
||
catch (OperationCanceledException)
|
||
{
|
||
throw;
|
||
}
|
||
catch (Exception ex)
|
||
{
|
||
// The connection opened, so the server is up — this query is what failed (an identifier that
|
||
// is not in the catalog, a lock, a permission). That is this group's problem alone.
|
||
_logger.LogWarning(
|
||
ex, "Sql poll group ({Model}, {Members} tag(s)) failed: {Sql}",
|
||
plan.Model, plan.Members.Count, plan.SqlText);
|
||
return GroupResult.Failed;
|
||
}
|
||
}
|
||
}
|
||
|
||
/// <summary>
|
||
/// Reads every row into memory, projected onto <see cref="SqlQueryPlan.SelectedColumns"/>. Row
|
||
/// fetching stays on the async path so a result set that streams slowly still honours the deadline
|
||
/// token mid-read, rather than only at the ExecuteReader boundary.
|
||
/// </summary>
|
||
private async Task<GroupResult> MaterialiseAsync(
|
||
SqlQueryPlan plan, DbDataReader reader, CancellationToken cancellationToken)
|
||
{
|
||
var selected = plan.SelectedColumns;
|
||
var ordinals = new int[selected.Count];
|
||
var typeNames = new string[selected.Count];
|
||
var columnIndex = new Dictionary<string, int>(selected.Count, StringComparer.Ordinal);
|
||
for (var c = 0; c < selected.Count; c++)
|
||
{
|
||
// GetOrdinal rather than the position in SelectedColumns: the list is documented to be in
|
||
// ordinal order, but resolving it against the live result set is what makes a wrong slice
|
||
// impossible rather than merely unlikely.
|
||
ordinals[c] = reader.GetOrdinal(selected[c]);
|
||
typeNames[c] = SafeTypeName(reader, ordinals[c]);
|
||
columnIndex[selected[c]] = c;
|
||
}
|
||
|
||
var rows = new List<object?[]>();
|
||
while (await reader.ReadAsync(cancellationToken).ConfigureAwait(false))
|
||
{
|
||
var row = new object?[selected.Count];
|
||
for (var c = 0; c < selected.Count; c++)
|
||
row[c] = reader.IsDBNull(ordinals[c]) ? null : reader.GetValue(ordinals[c]);
|
||
rows.Add(row);
|
||
}
|
||
|
||
return GroupResult.Ok(rows, columnIndex, typeNames);
|
||
}
|
||
|
||
/// <summary>Reads a column's provider type name, falling back to the dialect's unknown-type default.</summary>
|
||
private static string SafeTypeName(DbDataReader reader, int ordinal)
|
||
{
|
||
try
|
||
{
|
||
return reader.GetDataTypeName(ordinal) ?? string.Empty;
|
||
}
|
||
catch (Exception ex) when (ex is NotSupportedException or InvalidOperationException
|
||
or IndexOutOfRangeException)
|
||
{
|
||
// A provider that declines to report a type name must not fail the poll — the tag's declared
|
||
// type (or the dialect's String fallback) covers it.
|
||
return string.Empty;
|
||
}
|
||
}
|
||
|
||
/// <summary>Maps one group's outcome onto every slot its members occupy in the caller's list.</summary>
|
||
private void Slice(
|
||
SqlQueryPlan plan,
|
||
GroupResult outcome,
|
||
Dictionary<SqlTagDefinition, List<int>> slots,
|
||
DataValueSnapshot[] results,
|
||
DateTime readAt)
|
||
{
|
||
// KeyValue indexes rows by the key column; WideRow's members all read the one selected row, so the
|
||
// where-column is not even in the result set (design §5.3). Both selections happen ONCE per group —
|
||
// they are group-wide facts, and doing them per member would report a contract violation N times.
|
||
var byKey = outcome.Succeeded && plan.Model == SqlTagModel.KeyValue
|
||
? IndexByKey(plan, outcome)
|
||
: null;
|
||
var wideRow = outcome.Succeeded && plan.Model == SqlTagModel.WideRow
|
||
? SelectWideRow(plan, outcome)
|
||
: null;
|
||
|
||
foreach (var member in plan.Members)
|
||
{
|
||
// Members may repeat (two tags on one keyValue, or the same ref twice) — every occurrence is fed.
|
||
if (!slots.TryGetValue(member, out var positions)) continue;
|
||
|
||
var snapshot = outcome.Succeeded
|
||
? MapMember(plan, member, outcome, byKey, wideRow, readAt)
|
||
: new DataValueSnapshot(null, outcome.FailureStatus, null, readAt);
|
||
|
||
foreach (var position in positions) results[position] = snapshot;
|
||
}
|
||
}
|
||
|
||
/// <summary>
|
||
/// Builds the key → row index a key-value plan slices through. Later rows win, so a source that
|
||
/// breaks the one-row-per-key contract (design §3.6) still yields a deterministic value; the
|
||
/// violation is reported through a rate-limited warning rather than silently.
|
||
/// </summary>
|
||
private Dictionary<string, object?[]> IndexByKey(SqlQueryPlan plan, GroupResult outcome)
|
||
{
|
||
var index = new Dictionary<string, object?[]>(StringComparer.Ordinal);
|
||
if (plan.KeyColumn is null || !outcome.ColumnIndex.TryGetValue(plan.KeyColumn, out var keyAt))
|
||
return index;
|
||
|
||
var duplicated = 0;
|
||
foreach (var row in outcome.Rows)
|
||
{
|
||
// A NULL key cell indexes nothing: Convert.ToString would fold it to "", which would then be
|
||
// matched by a tag whose keyValue is legitimately the empty string.
|
||
if (row[keyAt] is not { } keyCell) continue;
|
||
|
||
// Stringified because the bound keyValue is authored text while the column may be any type
|
||
// (an INTEGER station id, say) — the provider coerces on the way in, so match on the way out.
|
||
var key = Convert.ToString(keyCell, CultureInfo.InvariantCulture);
|
||
if (key is null) continue;
|
||
if (!index.TryAdd(key, row))
|
||
{
|
||
index[key] = row;
|
||
duplicated++;
|
||
}
|
||
}
|
||
|
||
if (duplicated > 0 && ShouldWarnAboutContract())
|
||
{
|
||
_logger.LogWarning(
|
||
"Sql poll source '{Table}' returned more than one row for {Count} key(s) in one poll; the " +
|
||
"last row wins. The key-value model requires one row per key — point the tag at a " +
|
||
"current-values table or an operator-authored latest-per-key view.",
|
||
plan.Members[0].Table, duplicated);
|
||
}
|
||
|
||
return index;
|
||
}
|
||
|
||
/// <summary>
|
||
/// Picks the single row every member of a wide-row plan reads, and reports an ambiguous selector.
|
||
/// <para><b>Last row wins</b> — the same tie-break <see cref="IndexByKey"/> applies, so both models
|
||
/// degrade the same way. It is deterministic only <em>within one result set</em>: the where-pair form
|
||
/// emits no <c>ORDER BY</c>, so across polls the "last" row is whatever the server returned last,
|
||
/// i.e. storage order. That is why more than one row is a reported contract violation and not merely
|
||
/// a tie to break — the model's premise (design §5.3) is that the selector identifies one row.</para>
|
||
/// <para><b>Why the where-pair form is deliberately NOT given the dialect's single-row limit.</b> A
|
||
/// <c>TOP 1</c> with no <c>ORDER BY</c> is not deterministic either — it returns whichever row the
|
||
/// plan happens to produce first, so it would trade one arbitrary row for a differently arbitrary
|
||
/// one. Worse, it would destroy the only evidence this method has that the selector is ambiguous
|
||
/// (<c>Rows.Count > 1</c>), converting a loud misconfiguration into a permanently silent one —
|
||
/// precisely the failure mode this class's summary calls its top risk. The <c>topByTimestamp</c>
|
||
/// form keeps its row limit because there the <c>ORDER BY</c> makes "the first row" a defined
|
||
/// answer. The accepted cost is that an ambiguous selector materialises every matching row for one
|
||
/// poll; the warning is what gets it fixed.</para>
|
||
/// </summary>
|
||
private object?[]? SelectWideRow(SqlQueryPlan plan, GroupResult outcome)
|
||
{
|
||
if (outcome.Rows.Count == 0) return null;
|
||
if (outcome.Rows.Count > 1 && ShouldWarnAboutContract())
|
||
{
|
||
var first = plan.Members[0];
|
||
var selector = string.IsNullOrWhiteSpace(first.RowSelectorColumn)
|
||
? string.Concat("topByTimestamp ", first.RowSelectorTopByTimestamp)
|
||
: string.Concat(first.RowSelectorColumn, " = ", first.RowSelectorValue);
|
||
_logger.LogWarning(
|
||
"Sql poll source '{Table}' returned {Rows} rows for the wide-row selector '{Selector}' in " +
|
||
"one poll; the last row read wins, and the query carries no ORDER BY, so which row that is " +
|
||
"depends on storage order. The wide-row model requires a selector matching exactly one row " +
|
||
"— narrow the whereColumn/whereValue pair, or point the tag at a latest-row view.",
|
||
first.Table, outcome.Rows.Count, selector);
|
||
}
|
||
|
||
return outcome.Rows[^1];
|
||
}
|
||
|
||
/// <summary>Maps one member's cell to a snapshot: value, quality, and source timestamp.</summary>
|
||
private DataValueSnapshot MapMember(
|
||
SqlQueryPlan plan,
|
||
SqlTagDefinition member,
|
||
GroupResult outcome,
|
||
Dictionary<string, object?[]>? byKey,
|
||
object?[]? wideRow,
|
||
DateTime readAt)
|
||
{
|
||
object?[]? row;
|
||
string? valueColumn;
|
||
string? timestampColumn;
|
||
if (plan.Model == SqlTagModel.KeyValue)
|
||
{
|
||
valueColumn = plan.ValueColumn;
|
||
timestampColumn = plan.TimestampColumn;
|
||
row = member.KeyValue is { } key && byKey is not null ? byKey.GetValueOrDefault(key) : null;
|
||
}
|
||
else
|
||
{
|
||
valueColumn = member.ColumnName;
|
||
// A wide-row plan's members may legitimately carry DIFFERENT timestamp columns (all of them are
|
||
// selected), which is why the plan-level TimestampColumn is null for this model by design.
|
||
timestampColumn = member.TimestampColumn;
|
||
row = wideRow;
|
||
}
|
||
|
||
// No row: the key was deleted, or the row selector matched nothing. Distinct from a NULL cell.
|
||
if (row is null) return new DataValueSnapshot(null, SqlStatusCodes.BadNoData, null, readAt);
|
||
|
||
// A timestamp that will not parse falls back to the poll clock rather than poisoning a good value.
|
||
var sourceTimestamp = ReadTimestamp(outcome, row, timestampColumn) ?? readAt;
|
||
|
||
if (valueColumn is null || !outcome.ColumnIndex.TryGetValue(valueColumn, out var valueAt))
|
||
{
|
||
// The planner selects every member's value column, so this is unreachable short of plan/result
|
||
// drift — loud rather than silently publishing someone else's cell.
|
||
_logger.LogError(
|
||
"Sql tag '{Tag}' reads column '{Column}', which the executed plan did not select.",
|
||
member.Name, valueColumn);
|
||
return new DataValueSnapshot(null, SqlStatusCodes.BadInternalError, sourceTimestamp, readAt);
|
||
}
|
||
|
||
var cell = row[valueAt];
|
||
if (cell is null)
|
||
{
|
||
return new DataValueSnapshot(
|
||
null, _nullIsBad ? SqlStatusCodes.Bad : SqlStatusCodes.Uncertain, sourceTimestamp, readAt);
|
||
}
|
||
|
||
var target = member.DeclaredType ?? _dialect.MapColumnType(outcome.TypeNames[valueAt]);
|
||
if (!TryCoerce(cell, target, out var value))
|
||
{
|
||
_logger.LogWarning(
|
||
"Sql tag '{Tag}' could not coerce a {Actual} cell to its {Declared} type.",
|
||
member.Name, cell.GetType().Name, target);
|
||
return new DataValueSnapshot(null, SqlStatusCodes.BadTypeMismatch, sourceTimestamp, readAt);
|
||
}
|
||
|
||
return new DataValueSnapshot(value, SqlStatusCodes.Good, sourceTimestamp, readAt);
|
||
}
|
||
|
||
/// <summary>Reads a row's source timestamp, or null when there is no column, no value, or no parse.</summary>
|
||
private static DateTime? ReadTimestamp(GroupResult outcome, object?[] row, string? timestampColumn)
|
||
{
|
||
if (string.IsNullOrWhiteSpace(timestampColumn)) return null;
|
||
if (!outcome.ColumnIndex.TryGetValue(timestampColumn, out var at)) return null;
|
||
if (row[at] is not { } cell) return null;
|
||
return TryCoerce(cell, DriverDataType.DateTime, out var value) ? (DateTime?)value : null;
|
||
}
|
||
|
||
/// <summary>
|
||
/// Coerces a provider cell to the tag's effective <see cref="DriverDataType"/>, so the published CLR
|
||
/// type matches the type the address space declared for the node. Never throws.
|
||
/// </summary>
|
||
private static bool TryCoerce(object cell, DriverDataType target, out object? value)
|
||
{
|
||
try
|
||
{
|
||
value = target switch
|
||
{
|
||
DriverDataType.Boolean => Convert.ToBoolean(cell, CultureInfo.InvariantCulture),
|
||
DriverDataType.Int16 => Convert.ToInt16(cell, CultureInfo.InvariantCulture),
|
||
DriverDataType.Int32 => Convert.ToInt32(cell, CultureInfo.InvariantCulture),
|
||
DriverDataType.Int64 => Convert.ToInt64(cell, CultureInfo.InvariantCulture),
|
||
DriverDataType.UInt16 => Convert.ToUInt16(cell, CultureInfo.InvariantCulture),
|
||
DriverDataType.UInt32 => Convert.ToUInt32(cell, CultureInfo.InvariantCulture),
|
||
DriverDataType.UInt64 => Convert.ToUInt64(cell, CultureInfo.InvariantCulture),
|
||
DriverDataType.Float32 => Convert.ToSingle(cell, CultureInfo.InvariantCulture),
|
||
// decimal/numeric collapse here, with the documented precision caveat (design §8.6).
|
||
DriverDataType.Float64 => Convert.ToDouble(cell, CultureInfo.InvariantCulture),
|
||
DriverDataType.DateTime => ToUtc(cell),
|
||
_ => Convert.ToString(cell, CultureInfo.InvariantCulture) ?? string.Empty,
|
||
};
|
||
return true;
|
||
}
|
||
catch (Exception ex) when (ex is InvalidCastException or FormatException
|
||
or OverflowException or ArgumentException)
|
||
{
|
||
value = null;
|
||
return false;
|
||
}
|
||
}
|
||
|
||
/// <summary>
|
||
/// Normalises a timestamp cell to UTC. A naive <c>datetime</c> (no offset — the common SQL Server
|
||
/// shape) is <b>assumed</b> to already be UTC rather than reinterpreted through the server's local
|
||
/// zone, which would silently shift every source timestamp by the host's offset.
|
||
/// </summary>
|
||
private static DateTime ToUtc(object cell) => cell switch
|
||
{
|
||
DateTime { Kind: DateTimeKind.Utc } utc => utc,
|
||
DateTime { Kind: DateTimeKind.Local } local => local.ToUniversalTime(),
|
||
DateTime unspecified => DateTime.SpecifyKind(unspecified, DateTimeKind.Utc),
|
||
DateTimeOffset offset => offset.UtcDateTime,
|
||
string text => DateTime.Parse(
|
||
text, CultureInfo.InvariantCulture,
|
||
DateTimeStyles.AssumeUniversal | DateTimeStyles.AdjustToUniversal),
|
||
_ => DateTime.SpecifyKind(
|
||
Convert.ToDateTime(cell, CultureInfo.InvariantCulture), DateTimeKind.Utc),
|
||
};
|
||
|
||
/// <summary>Rate-limits the source-contract warning so a misconfigured table cannot flood the log.</summary>
|
||
private bool ShouldWarnAboutContract()
|
||
{
|
||
var now = Stopwatch.GetTimestamp();
|
||
var last = Interlocked.Read(ref _lastContractWarningTicks);
|
||
if (last != 0 && Stopwatch.GetElapsedTime(last, now) < ContractWarningInterval) return false;
|
||
return Interlocked.CompareExchange(ref _lastContractWarningTicks, now, last) == last;
|
||
}
|
||
|
||
/// <summary>One group's outcome: either a materialised result set, or the status its members inherit.</summary>
|
||
private sealed class GroupResult
|
||
{
|
||
private static readonly IReadOnlyList<object?[]> NoRows = [];
|
||
private static readonly IReadOnlyDictionary<string, int> NoColumns =
|
||
new Dictionary<string, int>(StringComparer.Ordinal);
|
||
|
||
private GroupResult(
|
||
uint failureStatus,
|
||
IReadOnlyList<object?[]> rows,
|
||
IReadOnlyDictionary<string, int> columnIndex,
|
||
IReadOnlyList<string> typeNames)
|
||
{
|
||
FailureStatus = failureStatus;
|
||
Rows = rows;
|
||
ColumnIndex = columnIndex;
|
||
TypeNames = typeNames;
|
||
}
|
||
|
||
/// <summary>The group exceeded its wall-clock deadline (design §8.3).</summary>
|
||
public static GroupResult TimedOut { get; } =
|
||
new(SqlStatusCodes.BadTimeout, NoRows, NoColumns, []);
|
||
|
||
/// <summary>The connection was fine but the query was not.</summary>
|
||
public static GroupResult Failed { get; } =
|
||
new(SqlStatusCodes.BadCommunicationError, NoRows, NoColumns, []);
|
||
|
||
/// <summary>The status every member inherits when <see cref="Succeeded"/> is false.</summary>
|
||
public uint FailureStatus { get; }
|
||
|
||
/// <summary>The materialised rows, projected onto the plan's selected columns.</summary>
|
||
public IReadOnlyList<object?[]> Rows { get; }
|
||
|
||
/// <summary>Selected column name → its position within each row array.</summary>
|
||
public IReadOnlyDictionary<string, int> ColumnIndex { get; }
|
||
|
||
/// <summary>Each selected column's provider type name, for dialect type inference.</summary>
|
||
public IReadOnlyList<string> TypeNames { get; }
|
||
|
||
/// <summary>True when the query ran; a zero-row result set is still a success.</summary>
|
||
public bool Succeeded => FailureStatus == SqlStatusCodes.Good;
|
||
|
||
/// <summary>A completed result set.</summary>
|
||
/// <param name="rows">The materialised rows.</param>
|
||
/// <param name="columnIndex">Selected column name → row-array position.</param>
|
||
/// <param name="typeNames">Each selected column's provider type name.</param>
|
||
/// <returns>A successful outcome.</returns>
|
||
public static GroupResult Ok(
|
||
IReadOnlyList<object?[]> rows,
|
||
IReadOnlyDictionary<string, int> columnIndex,
|
||
IReadOnlyList<string> typeNames)
|
||
=> new(SqlStatusCodes.Good, rows, columnIndex, typeNames);
|
||
}
|
||
}
|
||
|
||
/// <summary>
|
||
/// The OPC UA status codes the Sql driver publishes, and the quality-class predicates its tests read
|
||
/// them back with.
|
||
/// <para><b>Why a driver-local table rather than a shared one.</b> Every driver in this tree carries its
|
||
/// own (<c>TwinCATStatusMapper</c>, <c>AbCipStatusMapper</c>, <c>FocasStatusMapper</c>,
|
||
/// <c>AbLegacyStatusMapper</c>, Galaxy's <c>StatusCodeMap</c>) — there is no shared helper in
|
||
/// <c>Core.Abstractions</c>, because drivers deliberately do not reference the OPC UA SDK and
|
||
/// <see cref="DataValueSnapshot.StatusCode"/> is a bare <see cref="uint"/>. The values below were read
|
||
/// off <c>Opc.Ua.StatusCodes</c> rather than copied from a sibling driver, since two of those tables
|
||
/// carry transcription errors.</para>
|
||
/// </summary>
|
||
public static class SqlStatusCodes
|
||
{
|
||
/// <summary>The value is good. (<c>Good</c>)</summary>
|
||
public const uint Good = 0x00000000u;
|
||
|
||
/// <summary>Quality is uncertain with no more specific reason — a NULL cell under <c>nullIsBad=false</c>.</summary>
|
||
public const uint Uncertain = 0x40000000u;
|
||
|
||
/// <summary>Quality is bad with no more specific reason — a NULL cell under <c>nullIsBad=true</c>.</summary>
|
||
public const uint Bad = 0x80000000u;
|
||
|
||
/// <summary>
|
||
/// No row was returned for this tag — the key is absent from the result set, or the wide-row
|
||
/// selector matched nothing. <b>Deliberately distinct from a NULL cell:</b> the row is gone, versus
|
||
/// the row is there and the value is not.
|
||
/// </summary>
|
||
public const uint BadNoData = 0x809B0000u;
|
||
|
||
/// <summary>The group exceeded its client-side wall-clock deadline (design §8.3).</summary>
|
||
public const uint BadTimeout = 0x800A0000u;
|
||
|
||
/// <summary>The reference resolves to no authored tag.</summary>
|
||
public const uint BadNodeIdUnknown = 0x80340000u;
|
||
|
||
/// <summary>The query failed after the connection opened.</summary>
|
||
public const uint BadCommunicationError = 0x80050000u;
|
||
|
||
/// <summary>The cell could not be coerced to the tag's effective data type.</summary>
|
||
public const uint BadTypeMismatch = 0x80740000u;
|
||
|
||
/// <summary>A tag definition the planner rejected — an authoring fault, not a database fault.</summary>
|
||
public const uint BadConfigurationError = 0x80890000u;
|
||
|
||
/// <summary>A defect in the reader itself; never expected in the field.</summary>
|
||
public const uint BadInternalError = 0x80020000u;
|
||
|
||
/// <summary>The quality-class mask — the top two bits of an OPC UA status code.</summary>
|
||
private const uint SeverityMask = 0xC0000000u;
|
||
|
||
/// <summary>True when <paramref name="statusCode"/> is in the Good quality class.</summary>
|
||
/// <param name="statusCode">The status code to classify.</param>
|
||
/// <returns>True when the code's severity bits are Good.</returns>
|
||
public static bool IsGood(uint statusCode) => (statusCode & SeverityMask) == 0u;
|
||
|
||
/// <summary>True when <paramref name="statusCode"/> is in the Uncertain quality class.</summary>
|
||
/// <param name="statusCode">The status code to classify.</param>
|
||
/// <returns>True when the code's severity bits are Uncertain.</returns>
|
||
public static bool IsUncertain(uint statusCode) => (statusCode & SeverityMask) == Uncertain;
|
||
|
||
/// <summary>True when <paramref name="statusCode"/> is in the Bad quality class.</summary>
|
||
/// <param name="statusCode">The status code to classify.</param>
|
||
/// <returns>True when the code's severity bits are Bad.</returns>
|
||
public static bool IsBad(uint statusCode) => (statusCode & SeverityMask) >= Bad;
|
||
}
|