lmxopcua/docs/drivers/Sql.md

SQL Poll Driver

Status lives in Gitea. Open follow-ups and known gaps noted in this doc are tracked as issues on gitea.dohertylan.com/dohertj2/lmxopcua (label migrated, 2026-11-06). This doc describes the system — do not track status here; file new work as an issue.

Getting-started guide for the Sql driver — a read-only Equipment-kind driver that polls rows out of a relational database and publishes them as OPC UA variables. For the design rationale read docs/plans/2026-07-15-sql-poll-driver-design.md; for the build-vs-plan record read docs/plans/2026-07-24-sql-poll-driver.md.

What it talks to

An ordinary SQL database — typically a MES / LIMS / historian staging table that some other system already writes to. There is no protocol here and nothing to "connect" to a machine: the driver runs SELECTs on a timer and turns cells into tag values.

All five providers construct: SqlServer (Microsoft.Data.SqlClient), Postgres (Npgsql), Oracle (ODP.NET managed — Oracle 12.2 or later, the floor the FETCH FIRST paging needs), MySql (MySqlConnector, MIT, fully managed — MySQL 8 / MariaDB) and Odbc (System.Data.Odbc, see below). An undeclared provider value fails the driver rather than silently falling back. The provider → dialect map is one shared table (SqlDialects) that the factory, the Test Connect probe and the browse picker all read, so the three can no longer disagree about what a build supports.

MySQL specifics: a MySQL database is the schema (schema.table = database.table; the default schema is the connection's Database), identifiers are backtick-quoted (64-character ceiling, no trailing space), and table names are case-sensitive on Linux — the catalog gate's spelling substitution is what lets PLANT.TAG_VALUES bind to plant.tag_values. TINYINT(1) reads as a boolean under MySqlConnector's default TreatTinyAsBoolean; BIT(n), TIME, JSON and binary types infer as String. Live gate: Docker/mysql/docker-compose.yml + MySqlReadTests (SQL_MYSQL_ENDPOINT / SQL_MYSQL_PASSWORD).

ODBC specifics: OdbcDialect is the ANSI SQL:2008 profile — double-quoted identifiers, positional ? parameters, the standard INFORMATION_SCHEMA views and FETCH FIRST n ROWS ONLY. ODBC abstracts the wire, not the SQL, so the backend behind the DSN must accept all four: PostgreSQL (psqlODBC — the live gate), Snowflake and similar do; SQL Server (needs OFFSET … FETCH after an ORDER BY), MySQL (no FETCH FIRST) and DB2 (no INFORMATION_SCHEMA) do not and should use their native provider. It is also the one provider that is not fully managed: every host running an Odbc instance needs the platform driver manager (unixODBC / iODBC / Windows' own) and the vendor's ODBC driver — on the driver nodes and, for browse, the AdminUI nodes. Parameters bind in order, so the planner, the catalog loader and the browse session all add them in marker order, and a named query's params bind in authored order (the factory refuses a statement whose ? count differs). Live gate: Docker/odbc/run-odbc-gate.sh runs OdbcReadTests in a throwaway Linux container with unixODBC + psqlODBC against a throwaway PostgreSQL; on macOS the suite is unit-tested only (OdbcDialectTests) because no vendor ODBC driver is installed.

Two provider-specific things are worth knowing before you author against them:

• Identifier case folds, and the catalog gate absorbs it. Postgres lower-cases and Oracle upper-cases unquoted identifiers. Every authored identifier is resolved to the catalog's own spelling, so an authored widgets binds to Oracle's WIDGETS rather than failing.
• Oracle NUMBER infers as Float64. Column metadata does not carry precision/scale to the type map, so an integer-shaped NUMBER still infers double. Use the per-tag type override (see Optional per-tag fields) when the OPC UA type matters.

Read-only, structurally. SqlDriver does not implement IWritable, so no configuration can produce a write. The allowWrites config field exists and is inert; the factory warns when it is authored true rather than failing, on the grounds that the flag cannot do harm but an operator who believes writes are enabled can. → #753

Tag models

A tag says how to find its value. Three models ship: KeyValue, WideRow and the named-query Query.

KeyValue — the EAV shape

One row per tag. The driver matches keyColumn = keyValue and reads valueColumn.

{
  "model": "KeyValue",
  "table": "dbo.TagValues",
  "keyColumn": "TagName",
  "keyValue": "Line1.Oven.Temp",
  "valueColumn": "Value",
  "timestampColumn": "UpdatedUtc"
}

WideRow — one row, many signals

The tag names its columnName; the row is picked either by a whereColumn/whereValue pair or by newest-timestamp.

⚠️ The selector keys live inside a nested rowSelector object, not at the top level. A blob that spells whereColumn / whereValue / topByTimestamp beside columnName carries no row selector as far as SqlEquipmentTagParser is concerned: the tag is refused, the driver skips it, and it reads BadNodeIdUnknown forever. The typed editor, the browse-commit mapper and the deploy gate all use the nested form.

{
  "model": "WideRow",
  "table": "dbo.LineStatus",
  "columnName": "OvenTemp",
  "rowSelector": { "whereColumn": "LineId", "whereValue": "L1" }
}

{
  "model": "WideRow",
  "table": "dbo.Readings",
  "columnName": "OvenTemp",
  "rowSelector": { "topByTimestamp": "CapturedUtc" }
}

⚠️ topByTimestamp needs an ordering column that is unique per row. It is executed as an ORDER BY <column> DESC with the dialect's single-row limit, so when the column ties the server returns an arbitrary row among the tied ones — and the driver cannot warn about it, because the other rows were discarded before the result set reached it (the whereColumn/whereValue form does warn, on Rows.Count > 1). A date-typed column ties for every row written on the same day, and browse-commit will nominate one. Prefer a datetime2/sequence column that is unique per row, or point the tag at a latest-row view.

Both value-bearing fields — whereValue here and keyValue on the KeyValue model — may be authored as a JSON string, number or boolean, and are captured as text. They are bound as parameters either way, and matched back as text (the key cell is stringified), so a key column that holds an INTEGER station id can be authored either "7" or 7. A JSON object, array or null is not a value and is refused. On PostgreSQL a string bind is sent untyped (NpgsqlDbType.Unknown, PostgresDialect.ConfigureParameter, STAB-24) so the server infers its type from the compared column; bound as text, a "7" against an integer key fails with operator does not exist: integer = text.

⚠️ Known limitation — key normalisation. Row slice-back matches the returned key cell by its text form (Convert.ToString, ordinal), while the server now matches by the column's inferred type. An authored key the server treats as equal but whose text differs therefore selects a row that the driver then cannot find, and the tag reads no-row: "007" against an integer column (returns 7), "1.50" against numeric (returns 1.5), and an upper-case GUID against a uuid column (returned lower-case). Author keys in the column's canonical text form. SQL Server behaves the same way. (They were read asymmetrically until 2026-08-10, so a numerically authored keyValue refused the tag while the identically-bound whereValue was served — and the deploy gate reported the key as missing when it was right there in the blob.)

Query — a named read-only query (#753)

For a join, a view you cannot create, or a computed projection. The driver config declares the statement under queries; the tag names the query, the result column it reads, and optionally the row:

{ "queries": {
    "activeBatch": {
      "sql": "SELECT line, speed, temp, ts FROM dbo.LineStatus WHERE active = @active",
      "params": { "active": true },
      "maxRows": 100
    } } }

{ "model": "Query", "query": "activeBatch", "column": "speed",
  "rowKey": { "column": "line", "value": "Line1" }, "timestampColumn": "ts", "type": "Float64" }

• One execution per query per poll, however many tags read it.
• rowKey picks the row (matched as text against that result column, like keyValue). Without a rowKey the query must return exactly one row. No matching row ⇒ BadNoData; two matching rows (or two rows with no rowKey) ⇒ BadConfigurationError — never "last row wins".
• Result column names (column, rowKey.column, timestampColumn) match exactly first, then by a unique case-insensitive match, so Postgres' lower-cased and Oracle's upper-cased aliases still bind. A column the query does not return ⇒ BadConfigurationError plus a rate-limited warning listing what it did return.
• maxRows (default 1000, at most 100 000) is enforced client-side: the reader stops one row past it and the whole query publishes BadConfigurationError. A statement cannot be wrapped in a provider row limit portably, so narrow the WHERE rather than relying on the cap.
• Parameters are always bound. Keys are bare names (active; a leading @/: is tolerated); values are JSON scalars or null. Write the provider's own marker in the SQL: @active on SqlServer, Postgres and MySql, :p_active on Oracle, a positional ? on Odbc (bound in authored order). The factory refuses a declared parameter whose marker the statement never uses.
• Read-only, twice over. SqlReadOnlyStatement refuses at construction (and in the AdminUI form) anything but a single SELECT/WITH statement, and any of INSERT UPDATE DELETE MERGE UPSERT DROP ALTER CREATE TRUNCATE EXEC EXECUTE CALL GRANT REVOKE DENY INTO COPY LOCK RENAME outside a literal or comment — so SELECT … INTO, FOR UPDATE and a data-modifying CTE are refused too. Behind that guard the statement runs inside a transaction the driver always rolls back, so a side effect the guard cannot see (a function that writes) is never committed. A non-transactional side effect (an e-mail procedure, a sequence) is outside both, so give the connecting principal SELECT only.
• A tag naming a query the driver config does not declare is skipped at Initialize (it reads BadNodeIdUnknown, and /hosts names the missing query). One unusable queries entry fails the driver instead, because every tag that names it would otherwise go dark unexplained.
• Query tags read no table, so they need no catalog gate — a driver serving only named queries skips the catalog load entirely. Nothing from a Query tag is ever quoted into SQL.

Optional per-tag fields

Field	Effect
timestampColumn	Source timestamp for the value. Absent ⇒ the driver stamps the poll time.
type	Explicit OPC UA type override. Absent ⇒ inferred from the result set's column metadata.

There is no per-tag poll-interval field. The interval is the subscriber's, floored by PollGroupEngine.DefaultMinInterval, falling back to the instance's defaultPollInterval when the subscriber names none. (A pollIntervalMs per-tag override was documented here for a while and never existed in the parser — nothing reads it.)

Driver config

{
  "provider": "SqlServer",
  "connectionStringRef": "MesStaging",
  "defaultPollInterval": "00:00:05",
  "operationTimeout": "00:00:15",
  "commandTimeout": "00:00:10",
  "maxConcurrentGroups": 4,
  "nullIsBad": false
}

Key	Default	Notes
provider	SqlServer	SqlServer, Postgres, Oracle, MySql or Odbc.
connectionStringRef	—	A name, never a connection string. See below.
defaultPollInterval	00:00:05	Applied to any group without its own interval.
operationTimeout	00:00:15	Client-side wall-clock deadline; a breach surfaces BadTimeout.
commandTimeout	00:00:10	ADO.NET server-side backstop. Must be less than operationTimeout — it is the backstop, not the deadline.
maxConcurrentGroups	4	Connection-pool guard: cap on group queries in flight.
nullIsBad	false	false ⇒ a NULL cell publishes Uncertain; true ⇒ Bad.
allowWrites	—	Inert. v1 has no write path; authoring true logs a warning.
queries	—	Named read-only queries for Query tags — see Query. Edited as JSON in the driver form, which applies the factory's own rules.

Credentials never enter the deployment artifact

connectionStringRef is a name resolved at Initialize from the environment / secret store — e.g. Sql__ConnectionStrings__MesStaging. There is no field that holds a connection string, so a sealed deployment artifact cannot carry database credentials even by accident.

A useful side effect of the in-place reinit added for #516: a rotated credential is picked up on the next config apply without a process restart, because the driver re-resolves the reference when it re-parses.

How polling works

Tags are grouped by source and one query is issued per group, not per tag — SqlGroupPlanner emits a SqlQueryPlan per distinct group key. Grouping is deterministic (groups appear in the input order of their first member, members keep their input order, parameters keep first-appearance order), so an unchanged configuration produces byte-identical SQL.

Every key-value and where-pair query is row-capped (R06:S-1). A key-value plan reads at most twice as many rows as it has distinct keys and asks for one more (TOP (2n+1) / LIMIT 2n+1 / FETCH FIRST 2n+1 ROWS ONLY); a where-pair wide-row plan asks for two. The reader stops there even if a provider ignores the limit.

• A duplicated key is refused per key. When a key comes back more than once, only the tags on THAT key publish BadConfigurationError; every other key in the group reads normally, whatever else is present or absent. (Before this, a duplicate was last-row-wins Good.)
• A gross excess fails the group. Receiving the extra row past the cap means a history/events table or a non-unique where column, which cannot be attributed because the rest is unread, so every member of the group publishes BadConfigurationError. For a where-pair any second row is this case.
• Each case logs a per-plan rate-limited warning naming the table and the selector (or an example key). topByTimestamp keeps its single-row limit.
• Upgrade note: a deployment whose key-value source carries stray duplicate rows, or whose where column is not unique, will see those tags go Bad where they used to read an arbitrary row at Good.

Every such refusal, like the other refusals (absent row, NULL cell, uncoercible cell), is the database answering: it advances LastSuccessfulRead and restores Healthy after a comms fault.

⚠️ Group keys are case-sensitive on identifiers. Two tags authored dbo.T and DBO.T plan as two groups — one extra round-trip, not a correctness problem, but worth knowing if a group count looks higher than expected.

SQL injection: how the driver is safe

The two classes of authored string are handled differently, and deliberately so:

• Value-bearing fields (keyValue, whereValue) are bound as DbParameters. They are never concatenated into command text.
• Identifier-bearing fields (table, keyColumn, valueColumn, timestampColumn, columnName, whereColumn, topByTimestamp) cannot be parameterised in SQL. They are validated against the live catalog — INFORMATION_SCHEMA on SQL Server and Postgres, the ALL_* views (ALL_USERS / ALL_TABLES ∪ ALL_VIEWS / ALL_TAB_COLUMNS) on Oracle (MySQL also uses information_schema) — and dialect-quoted before they reach a command.

The parser deliberately does not sanitise either kind — sanitising a value would corrupt legitimate data, and sanitising an identifier would mask an authoring error the catalog check reports properly.

⚠️ An unreadable catalog FAULTS the driver rather than rejecting every tag (Gitea #496). This is the safer failure: "I cannot verify these identifiers" and "these identifiers are invalid" are different conditions, and reporting the second when the first is true would send an operator hunting a configuration bug that does not exist. The catalog gate also substitutes the catalog's own spelling of an identifier, so a case difference between the authored blob and the database resolves rather than failing.

Browse

SqlDriverBrowser browses the live catalog — schema → table → column — so the /raw browse picker can commit columns as tags. A committed leaf carries schema, table and column: a column name alone cannot address a SQL tag, and an earlier version that emitted only the column produced tags whose typed editor opened empty.

Browse works against every constructed provider; the catalog query is the dialect's own (INFORMATION_SCHEMA on SQL Server, Postgres and MySQL — with pg_catalog / information_schema and MySQL's mysql / performance_schema / sys excluded — and the ALL_* views on Oracle).

The committed row selector (Gitea #552)

A browse-committed column becomes a WideRow tag, and a wide-row tag must say which row it reads — the table and column alone are not a binding. The browse session therefore states a fourth address field alongside schema/table/column: the table's preferred date/time-typed column, which the commit writes as rowSelector.topByTimestamp ("read the newest row"). Before this the commit emitted a selector-less blob that SqlEquipmentTagParser and the authoring gate both refuse, so no browse-committed Sql tag was ever readable.

Three properties of the choice are deliberate:

• Candidacy is by declared TYPE, never by column name. The session asks ISqlDialect.MapColumnType for DriverDataType.DateTime — the same metadata the browse tree already renders per column. A name heuristic (*_ts, *time*) would happily order a wide-row read by a string column called timestamp, binding the tag to whichever row sorts last lexically.
• Candidates are RANKED: a time-bearing type beats a date-only one. MapColumnType reports DriverDataType.DateTime for SQL Server's and PostgreSQL's date too, and ordering a single-row read by a date-only column ties for every row written on the same day — "the newest row" quietly becomes storage order among the ties, and the reader cannot warn about it because the server has already discarded the others. So a datetime2/timestamptz/… column wins over a date one wherever both exist, regardless of declaration order. A date-only column is still stated when it is all the table has, so such a table stays bindable by browse rather than becoming un-committable. Oracle's DATE is exempt — it carries a time-of-day component (which is why OracleDialect maps it faithfully rather than as a widening), so it is not demoted there.
• The tie-break within a rank is first in catalog order (every dialect's column query orders by ordinal position), so the pick is stable across calls, connections and servers. On a CreatedUtc/ModifiedUtc pair it takes the earlier-declared one. It is a visible, editable default: the typed editor shows the selector and the operator can change it — to the other timestamp, or to a whereColumn/whereValue pair entirely.

A table with no date/time column cannot be bound by browse, and the commit says so. The remaining selector is a whereColumn/whereValue pair whose value is data, not schema, so nothing in a catalog can supply it. RawBrowseCommitMapper.DescribeUncommittableLeaf refuses that leaf by name, names the table, and points at the tag editor; no selector is invented, and the mapper's blob stays selector-less rather than dropping to a generic {"address": …} the typed editor cannot read.

Authoring in the AdminUI

• Driver config — /raw → driver node → Configure. Sql is a single-connection driver: the connection lives on the driver, not on a device, and the modal says so.
• Tags — the typed SqlTagConfigModel editor, or CSV import (Sql has typed CSV columns), or browse-commit from the picker.

Capabilities

IDriver, ITagDiscovery, IReadable, ISubscribable, IHostConnectivityProbe.

No IWritable (structural, above). No IRediscoverable — a SQL schema change is not signalled, and DiscoverAsync echoes authored config rather than enumerating the database, so this driver is deliberately excluded from any discovery-diff machinery.

See also

• docs/Raw.md — the /raw authoring tree
• docs/drivers/README.md — the driver index
• Gitea #496 / #497 / #498 — the shipped follow-ups