The cross-driver README still described MQTT as pre-P2 — "Sparkplug B is P2
(`mode: SparkplugB` is a placeholder)" and "Sparkplug B has no fixture yet (P2)".
Both were false as of the P2 merge (c3a2b0f7): SparkplugB is a fully implemented
mode and the live suite is 15 tests (7 plain + 8 Sparkplug) against a
project-owned C# edge-node simulator. Mqtt.md and the scadaproj index were both
already correct; only this file missed the P2 sweep.
Filed the nine Known-gaps rows as tracking issues and linked them, so the table
stops being the only record of them:
#507 IRediscoverable is inert in v3 (platform gap — also Galaxy/TwinCAT/AbCip)
#508 write-through (IWritable) — NCMD/DCMD + plain publish
#509 a metric name containing '/' cannot be browse-committed
#510 _canRebirth captured at browse-open — reaped session still draws button
#511 Sparkplug primary-host STATE publishing (ActAsPrimaryHost)
#512 .Browser references .Driver — extract the client-options builder
#513 editor writes a meaningless payloadFormat into a Sparkplug blob
#514 Request-rebirth unarmable on a plant that birthed before the window
#515 DataSet / Template / PropertySet metric support
#507 is the one worth reading: it is a v3 platform gap, not an MQTT bug —
DriverHostActor.HandleDiscoveredNodes hard-returns, so every IRediscoverable
driver fires into a void and the edge gaining a point needs a redeploy to show up.
No code change; docs + issue links only.
Wave 1 of the driver-expansion program. The branch was complete and live-gated but
had never been merged; it sat 15 commits ahead and 50 behind master.
Purely additive behind the existing IModbusTransport seam — the application protocol
(function codes, codecs, planner, coalescing, write path, probe, materialisation,
HistoryRead) is identical across TCP and RTU; only the wire framing differs
([addr][PDU][CRC-16], no MBAP, no TxId). Zero changes to codecs/planner/health.
Selected by a new Transport config field (ModbusTransportMode: Tcp | RtuOverTcp),
routed through ModbusTransportFactory (throws on unknown mode), with the string-enum
guard on options/DTO/factory and a Transport selector on the AdminUI Modbus form.
ModbusSocketLifecycle was extracted from ModbusTcpTransport (behaviour-preserving)
and is composed by both transports.
Descoped by design: direct-serial (System.IO.Ports) and Modbus ASCII — RTU buses are
reached exclusively via a serial->Ethernet gateway. No per-tag config changes; the
existing per-tag UnitId already makes a multi-drop bus work.
Re-validated against current master at merge time (the branch's own gate predates 50
commits of master): no overlapping files, clean merge, solution builds 0 errors, and
Modbus 344/344, Modbus.Addressing 161/161, AdminUI 740/740, Runtime 507/507,
Configuration 131/131 all pass. Confirmed the Transport selector is still reachable
through master's current authoring path (DriverConfigModal -> ModbusDriverForm), so
the branch's live /run gate result still applies.
`GrpcPort` is a nullable column added by Phase 5. Every insert in the docker-dev
seed is INSERT-if-not-exists — the right shape for a re-runnable seed, but it means
a new nullable column never reaches rows an earlier version of the file created. On
a persisted volume all six rows kept NULL, central found no dial targets, and
`TelemetryDial:Mode=Grpc` connected to nothing. (`AkkaPort` escaped this only
because it is NOT NULL with a default of 4053.)
Seed now backfills `GrpcPort IS NULL AND CreatedBy = 'docker-dev-seed'` to 4056.
NULL only — it must not overwrite a port an operator set deliberately on a
long-lived dev volume, and NULL is the unambiguous "predates the column" marker.
Deliberately NOT doing the suggested EF data migration for real deployments. The
correct value is each node's own `Telemetry:GrpcListenPort`, which is
per-deployment configuration the database cannot derive; a migration could only
invent one, and a WRONG dial target is worse than the NULL the dialer already skips
explicitly with a throttled warning. The rig can be backfilled because there the
port is fixed by docker-compose.yml and therefore actually knowable.
`docs/Telemetry.md` gains an upgrade section stating the prerequisite, the symptoms
(the throttled skip log, LISTEN with no ESTABLISHED, the pill never going live) and
the UPDATE to run before flipping any node to Grpc — plus why there is no migration.
A deploy that triggers a FULL address-space rebuild clears `_alarmConditions`
and re-materialises every condition node fresh — inactive, acked, confirmed,
unshelved. The engine reloads its persisted state and re-derives Active from the
predicate, so it ends up correct; but there is no transition to report, so
`LoadAsync` yields `EmissionKind.None` and `OnEngineEmission` drops it. Nothing
writes the node.
Live-reproduced on the docker-dev rig against the pre-fix image: an alarm that
had fired and cleared but was still UNACKNOWLEDGED came back from a rebuild
reporting Acknowledged with Retain=false — so it disappeared from
ConditionRefresh entirely, while the engine still held Unacknowledged. An
operator's outstanding alarm silently vanishes from the alarm list on deploy.
`ScriptedAlarmHostActor.ReassertConditionNodes` closes it. After each load it
reads the new `ScriptedAlarmEngine.GetProjections()` and sends one
`AlarmStateUpdate` per alarm NOT in the Part 9 no-event position.
Load-bearing properties:
- Node-only. It sends `AlarmStateUpdate` and nothing else — never the `alerts`
topic, never the telemetry hub. `alerts` is the historization path, so a row
per alarm per deploy would append a duplicate historian/AVEVA record every
time anyone deploys. This is why it reads a projection rather than re-emitting:
`ScriptedAlarmProjection` carries no `EmissionKind`, so there is nothing for
the emission machinery — or a future refactor — to mistake for a transition.
The issue's sketch said `GetState(alarmId)` + `LoadedAlarmIds`; that alone
would have clobbered the node's severity and message, so the projection also
carries the definition's severity, the resolved message template, and the
last-observed worst-input quality.
- Only non-normal alarms. One in the no-event position already matches what the
materialise built, so a never-fired alarm stays byte-identical to a deploy
without this pass (including its Message, which the materialise seeds with the
display name), and an all-normal deploy writes nothing at all.
- Timestamp is the condition's own `LastTransitionUtc`, not the deploy instant.
- No duplicate Part 9 events: `WriteAlarmCondition`'s delta-gate compares against
the node's CURRENT state, so a re-assert onto a freshly materialised node is a
genuine delta (fires once — correct, the rebuild reset what clients see) while
one onto a surgically-preserved node is suppressed.
Tests: 6 new host tests + 6 new engine tests. Falsifiability checked by
disabling the call — 4 of the 6 host tests go red; the other 2 are absence
guards against an over-broad fix and pass either way by design.
Live gate (docker-dev, central-2, pre-fix image vs. rebuilt image):
- pre-fix: condition absent from ConditionRefresh after a rebuild;
- post-fix: present, Unacknowledged, Retain=True, stamped with its own
LastTransitionUtc rather than the restart instant;
- /alerts stayed empty across two re-asserts, with the panel proven live by a
real ACTIVATED transition immediately afterwards — no duplicate history.
Until now ISqlDialect.QuoteIdentifier was the SOLE defence for authored
table/column names: an identifier went from the TagConfig blob straight into a
command text, bracket-quoted. The residual risk was bounded — a hostile name is
quoted into one nonexistent object, the query fails after the connection opens,
and the tag Bad-codes — but "bounded" is not "filtered", and the design promised
a filter. Both ISqlDialect and SqlServerDialect carried a doc paragraph saying so.
SqlCatalogGate + SqlCatalogLoader now resolve every authored identifier against
the live catalog at Initialize, and REPLACE it with the catalog's own spelling.
The identifier text in an emitted poll query is therefore a string this driver
read back out of ListSchemas/ListTables/ListColumns — not operator input. Quoting
becomes the backstop it was documented to be.
Decisions worth knowing before touching this:
- Substitution, not just validation. Matching is exact-ordinal first, then a
UNIQUE case-insensitive hit: SQL Server's default collation is CI so case
variants have always worked, and rejecting them would break valid deployments.
An ambiguous CI match under a case-sensitive collation is refused rather than
guessed — picking one would publish another column's data under the operator's
node, which is worse than rejecting the tag.
- Charset check BEFORE catalog lookup. Each identifier goes through
QuoteIdentifier for its rejection rules first, so a name carrying a control or
Unicode format character is rejected WITHOUT its value being echoed into a log
line (Trojan-Source). A name that passes is safe to render, which is why
catalog-miss messages do name it — an operator hunting a typo has to see what
they wrote. Both halves are pinned by tests.
- A rejected tag keeps its node. It is dropped from the POLLED table but stays in
the AUTHORED table, so it still materializes and reads BadNodeIdUnknown —
§8.1's specified outcome. My first wiring dropped it from both, which deleted
the node instead; an existing test (ReinitializeAsync_recoversFromFaulted)
caught it. A status code can only be published by a node that exists, and a
missing address-space entry is far harder to diagnose than a Bad quality.
- An unreadable catalog FAULTS Initialize; it does not reject every tag. That is
the absence of evidence about the tags, not evidence against them — rejecting
all of them would serve a confidently-empty address space and send the operator
hunting typos that do not exist. Zero visible schemas is treated the same way,
because that is exactly what a missing GRANT looks like. Faulting lands
DriverInstanceActor in Reconnecting with its retry timer running.
- Bounded load: one schema list, one default-schema scalar, then one ListTables
per distinct authored schema and one ListColumns per distinct authored relation
— never a full catalog enumeration, and nothing after Initialize. Every
authored name reaches the catalog queries as a bound @schema/@table parameter,
so building the allow-list cannot itself be an injection vector.
- ISqlDialect gains DefaultSchemaSql ("SELECT SCHEMA_NAME()"). An unqualified
`TagValues` must resolve in whatever schema the SERVER reports; hardcoding dbo
would be a silent lie on any estate that maps service accounts to their own
default schema. It is a query, not a constant, because the answer is
per-connection.
- Accepted v1 limitation: a 3-part db.schema.table (or linked-server name)
addresses a catalog this connection cannot enumerate, so it cannot be
allow-listed and is rejected with a message pointing at the fix — expose the
data through a view in the connected database.
VerifyLivenessAsync's wall-clock pattern is extracted to RunBoundedAsync and
reused, so the new I/O gets the same R2-01 / STAB-14 protection rather than a
second hand-rolled copy. (It also fixes a latent bug in the extracted code: the
OperationCanceledException arm detached the wrong task.)
Tests: 24 pure gate tests + 9 end-to-end driver tests against the real SQLite
catalog + 3 new live tests against the real SQL Server on 10.100.0.35 (21 in that
suite now pass, exercising the real SELECT SCHEMA_NAME() + INFORMATION_SCHEMA
path). Verified falsifiable: bypassing the gate turns exactly the three rejection
assertions red and leaves the rest green.
SqlInjectionRegressionTests is deliberately NOT rewritten to expect
BadNodeIdUnknown. It drives SqlPollReader directly, below the gate, and pins the
quoting backstop on its own — defence in depth is only worth the name if each
layer holds independently. Rewriting those assertions would delete the backstop's
only coverage and leave the gate a single point of failure. Its scope note, which
said the gate does not exist, is updated to say why it stays where it is.
The Sql driver's "a pasted literal connection string is never persisted" guarantee was
enforced only on the READ path: SqlDriverConfigDto has no connectionString property and
UnmappedMemberHandling.Skip drops the key on deserialization. Nothing stopped it being
WRITTEN. Config blobs are schemaless JSON columns and the fallback authoring pattern for
a driver without a typed form is a raw-JSON textarea, so an operator pasting
{"connectionString":"Server=...;Password=..."} would put a live credential in the
ConfigDb — persisted, replicated to every node's artifact cache, readable by anyone with
config access — while the runtime silently ignored it and the driver failed to connect.
Discarding a secret on read is not the same as refusing to store it.
DraftValidator.ValidateSqlConnectionStringNotPersisted fails the deploy
(SqlConnectionStringPersisted) when a Sql-typed driver instance's DriverConfig, or the
DeviceConfig of any device beneath it, carries a top-level connectionString key.
- Both surfaces are checked because DeviceConfig is merged onto DriverConfig before the
DTO sees it — a credential pasted into the device blob is the identical leak.
- The key is matched case-insensitively: System.Text.Json binds ConnectionString to a
connectionString property by default, so an ordinal match would leave the obvious
bypass open.
- Only the top level is scanned; the DTO is flat, so a nested occurrence cannot bind and
is not the mistake this rule exists to catch.
- The message never echoes the value — validation errors reach the AdminUI, the deploy
log and the audit trail, and repeating the string there would leak the credential the
rule is refusing to store. Pinned by a test.
- Malformed/blank/non-object JSON never throws: a parse failure would take down every
other rule in the same pass.
16 tests. Verified falsifiable — with the rule disabled the 6 positive assertions fail
and the 10 negatives stay green, so none of them passes vacuously.
Design §8.2's "if an inline connectionString is ever permitted (dev convenience only),
the AdminUI redacts it" carve-out is withdrawn in the same change: redaction only hides a
credential that has already been written.
Not covered, and filed as a follow-up: ClusterNode.DriverConfigOverridesJson is a third
persistence surface for a driver config blob, but it is not part of DraftSnapshot, so
this validator cannot see it.
Every value verified against Opc.Ua.StatusCodes in the pinned SDK (1.5.378.106) by
reflection, not by copying siblings. All are client-visible: OPC UA clients branch on
status.
The three named in #497:
- Historian.Gateway SampleMapper "BadNoData" 0x800E0000 -> 0x809B0000
(0x800E0000 is BadServerHalted)
- Galaxy "BadTimeout" 0x800B0000 -> 0x800A0000
(0x800B0000 is BadServiceUnsupported)
- TwinCAT BadTypeMismatch 0x80730000 -> 0x80740000
(0x80730000 is BadWriteNotSupported)
BadTypeMismatch was wrong in three MORE drivers the issue did not name — FOCAS,
AbLegacy and AbCip carried the same 0x80730000. Every call site is a
FormatException/InvalidCastException conversion failure, so the name was right and
the value was wrong in all four.
Adding the reflection guard then surfaced nine further defects offline tests had
never checked:
- Galaxy GoodLocalOverride 0x00D80000 -> 0x00960000 (not a UA code at all)
- Galaxy UncertainLastUsableValue 0x40A40000 -> 0x40900000
- Galaxy UncertainSensorNotAccurate 0x408D0000 -> 0x40930000
- Galaxy UncertainEngineeringUnitsExceeded 0x408E0000 -> 0x40940000
- Galaxy UncertainSubNormal 0x408F0000 -> 0x40950000
- TwinCAT BadOutOfService 0x80BE0000 -> 0x808D0000 (was BadProtocolVersionUnsupported)
- TwinCAT BadInvalidState 0x80350000 -> 0x80AF0000 (was BadAttributeIdInvalid)
- AbCip/AbLegacy GoodMoreData 0x00A70000 -> 0x00A60000 (was GoodCommunicationEvent)
- Historian.Gateway GatewayQualityMapper Good_LocalOverride 0x00D80000 -> 0x00960000
Galaxy's whole Uncertain block read as though the OPC DA quality byte could be
shifted into the UA substatus position; it cannot — the substatuses are an unrelated
enumeration. GatewayQualityMapper already held the correct table, which is what the
Galaxy values are now reconciled against.
Guard: StatusCodeParityTests (Core.Abstractions.Tests, which already project-
references every driver) reflects over every `const uint` in the deployed
ZB.MOM.WW.OtOpcUa.Driver.*.dll whose name reads as a status code, and asserts it
equals Opc.Ua.StatusCodes.<name>. Discovery is by convention, so a new driver
assembly referenced by that project is covered with no edit here. It went red on all
nine defects above before the fixes and now checks 109 constants green.
Because reflection can only see a NAMED constant, two inline literals were hoisted so
the guard can reach them: Galaxy's BadTimeout/Bad into StatusCodeMap, and
GatewayQualityMapper's 15-entry table into a new HistorianStatusCodes. An inline
`0x800B0000u, // BadTimeout` at a call site is exactly how the Galaxy defect survived.
The drivers stay SDK-free by design; only the test project references Opc.Ua.Core,
as the oracle.
Also corrected: tests that pinned the wrong values (Galaxy StatusCodeMapTests,
SampleMapperTests, GatewayQualityMapperTests — all written from the same bad source),
a mislabelled comment in DriverInstanceActorTests, and stale constants in two design
docs, one of them the unexecuted MTConnect driver design that would have propagated
BadNoData's wrong value into a new driver.
The CLI's SnapshotFormatter value->name table was checked against the SDK and is
correct; no change needed.
Task 26 — the P2 milestone gate. Ran the live /run verification on an isolated
docker-dev rig (project otopcua-mqtt, ports 9210/4850-4851/14350, image built
from this branch) against the real Mosquitto TLS+auth broker and the C#
Sparkplug edge-node simulator on 10.100.0.35, and recorded the result.
The gate found three defects. Two are the same defect class the P1 gate found
twice — a hand-maintained AdminUI surface left behind by a driver-side feature —
and the third is a pre-existing cross-driver bug that only a Sparkplug flow
could surface.
1. CRITICAL, FIXED — MqttDriverForm still shipped its P1 Sparkplug PLACEHOLDER.
Switching Mode to SparkplugB rendered a "not available yet" notice and NO
Group ID field, so with Sparkplug ingest fully shipped there was still no
way to author a Sparkplug driver from the AdminUI at all. Sparkplug.GroupId
is the driver's entire subscription filter (spBv1.0/{GroupId}/#): blank ⇒
connected, Healthy, ingesting nothing. Now authors all five Sparkplug keys,
MERGES over the existing sub-object rather than replacing it (so a key a
newer driver adds inside it survives an older AdminUI), leaves it untouched
in Plain mode, and validates the group id as the topic segment it is.
Pinned by 8 MqttDriverFormModelTests cases, verified falsifiable — 8 RED
with the fix stubbed out.
2. PRE-EXISTING + CROSS-DRIVER, FIXED — every node label in the shared
DriverBrowseTree was an <a href="#">. In a Blazor Web App, blazor.web.js's
enhanced-navigation click interceptor resolves a bare "#" against
<base href="/">, so clicking ANY browse-tree node label navigated the whole
AdminUI to "/", tore down the circuit, and destroyed the hosting modal —
losing the browse session AND the tag selection. @onclick:preventDefault
does not help: it suppresses the browser's default action, not Blazor's own
interceptor. Labels are now <button type="button">. Dates to the component's
introduction (2026-05-28) and affects EVERY driver's picker; it surfaced
only now because choosing a rebirth scope is the first flow that requires
clicking a label rather than the ▶ toggle (always a button, always fine).
3. FIXED (mitigation) — the browse tree rendered exactly once, at open, and
never again, while OpenAsync returns as soon as the SUBSCRIBE is granted.
On Sparkplug that means it is almost always empty forever, because births
are never retained. Added a Refresh button that re-reads the root against
the SAME session (nothing reconnects, nothing observed is lost, the tag
selection is kept, only the armed rebirth scope is cleared).
Live gate result (all against the real broker + simulator, group OtOpcUaSim):
- driver authored end-to-end through the fixed MqttDriverForm; blob is
Mode:SparkplugB + Sparkplug.GroupId:OtOpcUaSim with enums as names
- browse: BROWSER OPEN chip, Sparkplug-only rebirth panel, tree renders
Group → EdgeNode → [Device] → Metric with the device folder Filler1 and
node-level metric leaves SIDE BY SIDE, and "Node Control/Rebirth" as ONE
leaf (the metric-name-is-not-a-topic-segment rule holding)
- rebirth: group scope names the group + the 32-node cap, Cancel published
NOTHING (simulator log unchanged); edge-node scope → "1 command published"
and the simulator logged "rebirth NCMD #1 received; republishing metadata";
group scope → "2 commands published", both edge nodes re-birthed; a device
and a metric both resolve UP to "edge node EdgeA"; the panel does NOT render
for a Plain MQTT device
- browse-commit wrote bindable tuples with no topic/address key —
{"groupId","edgeNodeId","metricName","deviceId"} for the device metric and
deviceId ABSENT (not blank) for the node-level one, datatypes inherited from
the birth (Int64 / Float)
- deployed (Sealed), and both nodes served live changing values through OPC UA
at the 2 s cadence — Good 0x00000000, no BadNodeIdUnknown
- death→STALE: stopping the simulator drove both nodes to 0x80000000 with an
empty value ~2 s later, and restarting it recovered them to Good on the new
birth (FillCount back to its birth-declared 1000, then counting)
- rediscovery fired exactly TWICE (once per authored scope: OtOpcUaSim/EdgeA
7 metrics, OtOpcUaSim/EdgeA/Filler1 3 metrics) and was then gated across
many subsequent births/rebirths — the anti-storm change gate working, and
EdgeB correctly filtered out as an unauthored scope
- tag editor: opens in SparkplugB (mode inference on reopen) with all four
descriptor fields populated; blank group → "A Sparkplug group ID is
required."; "Plant/1" rejected; "Node Control/Rebirth" ACCEPTED and saved
with the slash whole; dataType override persists and its key is ABSENT again
after selecting "(from birth certificate)"
Two things are NOT verified, and are recorded rather than claimed:
- Rediscovery is INERT in v3 and this is a platform gap, not an MQTT one:
nothing subscribes to IRediscoverable.OnRediscoveryNeeded and
DriverHostActor.HandleDiscoveredNodes hard-returns. A DBIRTH introducing a
metric does NOT change the OPC UA tree; redeploy. Verified from the driver's
log line only, exactly as far as it can be verified.
- A metric name containing "/" cannot be BROWSE-COMMITTED: the derived raw tag
Name is a RawPath segment and may not contain "/", so the spec-mandatory
"Node Control/Rebirth" is refused ("Row 3: Name must not contain '/'"). The
refusal is loud and all-or-nothing, never a silently mis-bound tag, and
Manual entry is the working path. Fixing it needs a name-sanitisation policy
with a collision answer, so it was recorded rather than guessed at.
Also left open + documented: an empty browse tree cannot arm a rebirth (the
scope comes from a tree click, and the session side already accepts a bare
{group}/{edgeNode} — only a UI path is missing); _canRebirth is captured at
browse-open; the tag editor injects the Plain-only payloadFormat key into a
Sparkplug blob (cosmetic — the factory routes on the tuple).
Suites: offline 1528 passed / 0 failed (581 Driver.Mqtt + 809 AdminUI + 138
Core.Abstractions); live 15/15 against the broker + simulator.
Docs: docs/drivers/Mqtt.md brought fully up to date for P2 (Sparkplug config
sub-object, both tag shapes, the ingest state machine, rediscovery, browse +
rebirth + Refresh, a Known-gaps table); Mqtt-Test-Fixture.md gains the simulator
and drops its "Sparkplug has no fixture" claims; infra/README.md §3 gains the
simulator row; CLAUDE.md gains the simulator endpoint facts; the tracking doc
marks MQTT/Sparkplug COMPLETE. docker-dev/docker-compose.mqtt.yml is the
isolated-rig overlay the gate ran on.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Two gaps the Task 23/24 review found, both blocking Task 26's live gate.
Gap 1 — browse-commit produced a silently dead MQTT tag. RawBrowseCommitMapper
had no `Mqtt` case, so a committed leaf fell through to the generic
`{"address": …}` key. Neither MqttTagDefinitionFactory entry point reads
`address`: the tag deployed clean and reported BadNodeIdUnknown forever, with
no signal at commit time. Predates Task 23 (Plain was affected too), but Task 23
built the Sparkplug metric tree precisely so an operator could browse and commit
a binding.
The address is a DESCRIPTOR, not a reference string, and it cannot be recovered
from the browse node id: `{group}/{node}[/{device}]::{metric}` where a metric
name legitimately contains `/` (`Node Control/Rebirth`) — the ambiguity
MetricSeparator's remarks already name. So the session STATES it, via a new
`AttributeInfo.AddressFields` seam, and the mapper reads it. Which keys are
emitted is also how the mapper learns Plain vs Sparkplug — the driver type
reaching it is just `Mqtt`, and the mode lives on the driver config. Key names
are single-sourced in the new `MqttTagConfigKeys` (producer, mapper, factory),
with the literals pinned by a test so a symmetric rename cannot silently unbind
already-persisted blobs. A leaf with no stated address is refused at commit in
words rather than committed dead.
Gap 2 — RequestRebirthAsync had no UI. Task 23 shipped it backend-only; Task 26
step 1 assumes the button, and Task 25's runbook notes the picker tree stays
empty until a birth lands, so it is the only way to fill it on demand. Added to
the /raw browse modal: scope = the clicked tree node (device/metric resolve up to
their edge node, group fans out and is refused whole past 32), an explicit
two-click confirm naming the resolved scope and its consequence, the outcome or
the refusal shown rather than swallowed. Offered only for a Sparkplug session
(new `IRebirthCapableBrowseSession.RebirthAvailable` — one session class serves
both modes, so a type test alone would draw a button that can only throw) and
only to a DriverOperator; the server-side gate remains the boundary.
Tests: MQTT 545 → 581, AdminUI 781 → 800. The round-trip tests feed the emitted
blob to the REAL factory and were falsified by mutating the emitted key name.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The P1 live gate had to insert MQTT driver config directly via SQL: DriverConfigModal and
DeviceModal both fell through to "No typed config form for driver type Mqtt" with no raw-JSON
fallback, so broker host/port/TLS/credentials were unauthorable from the AdminUI. Task 12 built
the *tag* editor; these are the driver + device surfaces.
MqttDriverForm authors the whole MqttDriverOptions connection surface (host/port/clientId,
TLS + CA pin, credentials, protocol version/clean-session/keep-alive, connect timeout, reconnect
backoffs, mode, maxPayloadBytes, and the Plain sub-object). The Sparkplug sub-object stays P2 —
a marked placeholder mirroring MqttTagConfigEditor's stub, with any existing sparkplug keys
preserved untouched.
The connection is authored on the DRIVER, not the device — unlike Modbus/S7/OpcUaClient and
contrary to what docs/drivers/Mqtt.md said. DriverDeviceConfigMerger merges a device's keys up
only when the driver has exactly ONE device, so a device-authored broker connection vanishes
silently the moment a second device is added. MqttDeviceForm is therefore informational (the
GalaxyDeviceForm shape) and round-trips DeviceConfig verbatim; it flags legacy connection keys
left on a device by a pre-form deployment, because those still win via merge-up.
Serialization goes through the ONE shared MqttJson.Options instance (Task 9's decision), never a
fifth per-form copy — pinned by an assertion that an ordinal-only reader CANNOT bind the emitted
blob, and by extending the fleet-wide DriverPageJsonConverterTests guard to resolve a form's
serializer from an explicit external-instance registry when it has no _jsonOpts field. Dropping
the converter from MqttJson.Options reddens 3 tests and leaves the symmetric round-trip green —
the Task 9 lesson, reproduced.
RawTags is stripped from the emitted blob (the deploy artifact owns it) and unknown top-level
keys survive a load->save. Validation mirrors the driver's own [Range] bounds inline, and
ToOptions() additionally clamps, so an ignored error still cannot persist a
connectTimeoutSeconds:0 driver-brick. A non-blank clientId raises the P1 live-gate warning
inline (fixed ids make redundant pair nodes evict each other while both report Healthy).
AdminUI 761/761 green (was 730), TWAE-clean; Driver.Mqtt 266/266 green.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Task 14 — the P1 milestone gate. Ran the live /run verification on a docker-dev
rig against the real Mosquitto TLS+auth fixture, and recorded the result.
The gate did its job: it found MQTT was unauthorable in production.
RawDriverTypeDialog's driver-type list is a hand-maintained array that nobody
added MQTT to, so the /raw "New driver" picker never offered it — with every
other layer (contracts, driver, browser, factory, host registration, typed tag
editor) complete and 266 green unit tests. Nothing tied that array to
DriverTypeNames and AdminUI has no bUnit, so no offline test could see it.
Fixed, plus RawDriverTypeDialogParityTests: a reflection parity guard that
fails for ANY DriverTypeNames entry missing from the picker. Verified
falsifiable — RED with "…cannot be authored from the /raw New-driver picker,
so they are unreachable in production: Mqtt" before the one-line fix.
A second gap is left OPEN and documented, not fixed (no task ever covered it):
MqttDriverForm / MqttDeviceForm do not exist, and DriverConfigModal/DeviceModal
have no raw-JSON fallback — so an operator can create an MQTT driver but cannot
author its broker endpoint or credentials. P1 is not operator-complete until
those two razor forms land. The gate authored config via SQL to get past it.
Live gate result (isolated 2-node MAIN stack, image built from this branch,
fixture at 10.100.0.35:8883, AllowUntrustedServerCertificate rather than
pinning the CA into the rig):
- typed MQTT tag editor renders and round-trips; Json shows the JSON-path
field, Raw/Scalar hide it; wildcard topic a/+/c blocked at save; data-type
list offers Float64 and no Double; qos/retainSeed absent on "(driver
default)" and qos:2 written as a number; isHistorized/historianTagname
survive a topic-only edit; SparkplugB renders the P2 placeholder and
switching back keeps Plain values; jsonPath is guidance not a gate in all
three shapes ($ seeded only for a brand-new blank tag, blank stays absent,
clearing saves fine)
- deployed, and both editor-written blobs resolved and served changing live
values through OPC UA (22.8 / 1629, StatusCode Good) — no BadNodeIdUnknown
- the bespoke #-observation browser drove a real broker session and rendered
the fixture's actual topic tree (line1/{counter,state,temperature},
noise/chatter, retained/seed); a Modbus device's picker still correctly
reports "Browsing unavailable"
Also found live and documented (not a code change): both nodes of a redundant
pair run the driver, so a FIXED clientId makes them evict each other forever —
the broker logs "already connected, closing old connection" and both nodes
reconnect every ~2s while still reporting Healthy. Unset (the default) is
correct; confirmed 0 reconnects/60s on both nodes after removing it.
Suites: offline 1134 passed / 0 failed (266 Driver.Mqtt + 730 AdminUI incl. the
3 new guards + 138 Core.Abstractions); live 7/7 against the fixture.
Docs: new docs/drivers/Mqtt.md + Mqtt-Test-Fixture.md (the per-driver + fixture
convention every other driver follows), MQTT rows in docs/drivers/README.md,
TestConnectProbes.md, root README.md, and infra/README.md §3.
CLAUDE.md drift corrected while adding the MQTT endpoints:
- the "every fixture carries project: lmxopcua" claim was false — only the
MQTT fixture does; the filter returns one stack, not the fleet
- lmxopcua-fix.ps1 is Windows-VM-only; on macOS drive the host over ssh/rsync
(and the docker-dev rig itself runs locally under OrbStack)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Per-plan copy-paste command (subagent-driven-development in a git worktree),
plus the slash-command and executing-plans/resume alternatives, and a note that
the four independent plans can run in concurrent worktrees.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Add a living progress tracker (docs/plans/2026-07-24-driver-expansion-tracking.md)
for the driver-expansion program's Waves 0-2, linked from the program design doc's
document map. One row per deliverable pointing at design + implementation plan + status.
Add executable implementation plans (plan .md + co-located .tasks.json) for the four
Wave 1/2 drivers, each authored by writing-plans methodology off its design doc and
mirroring the relevant existing driver:
- Modbus RTU 11 tasks (extends Driver.Modbus; RTU-over-TCP, direct-serial descoped)
- SQL poll 22 tasks (ISqlDialect seam, SQL Server only in v1; SQLite + central-SQL fixtures)
- MTConnect Agent 23 tasks (browse free via Wave-0 seam; TrakHound-vs-hand-rolled = Task 0)
- MQTT/Sparkplug B 27 tasks (P1 plain MQTT 0-14 then P2 Sparkplug 15-26; MQTTnet-5 pinning spike = Task 0)
Wave 0 (universal browser) remains merged (056887d6) with its live gate open (#468).
All four new plans are design-only -> plan-ready; nothing built yet.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The simultaneous-cold-start split-brain, carried as a Phase 7 "candidate follow-up,
not shipped", is now closed by the opt-in Cluster:BootstrapGuard (279d1d0f). Documents
the guard in CLAUDE.md's bootstrap section and marks the Phase 7 finding closed.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Closes the residual Phase-7 gap: when BOTH nodes of a 2-node pair cold-start at the
same instant, each self-first seed runs FirstSeedNodeProcess, times out waiting for
the other, and forms its own 1-node cluster — two Primaries in one pair (the Phase 6
live gate reproduced this reliably on docker). The prior mitigation was operational
(staggered start / compose depends_on), which does not exist on production hardware.
The guard (dark switch Cluster:BootstrapGuard:Enabled, default OFF) prevents the split
without giving up cold-start-alone:
- The lower-address node is the preferred founder: self-first, forms immediately.
- The higher node probes its partner's Akka port (TCP connect) up to PartnerProbeSeconds:
reachable => peer-first (join the founder, never race it); unreachable => self-first
(partner is dead, form alone). The order is decided BEFORE the single JoinSeedNodes,
from an explicit reachability signal — never re-formed mid-handshake (the retired
SelfFormAfter failure mode). When on, Akka gets no config seeds (BuildClusterOptions)
and ClusterBootstrapCoordinator drives the join.
Review-driven hardening: case-insensitive tie-break (a hostname-casing mismatch would
reopen the split); fail-fast validation of the timing knobs; the residual "founder dies
in the probe->join window" hang is documented and made operator-visible (warning + a
restart recovers). Real-ActorSystem coordinator tests cover the load-bearing higher-node
cold-start-alone case; 147/147 Cluster tests pass.
Live-gated on docker-dev (site-a = enablement demo, serialization removed, guard on;
site-b keeps depends_on-serialization + guard off as the A/B control): simultaneous
start -> site-a-1 founds, site-a-2 probes-reachable-joins -> 250/240, NO split;
higher-node-alone -> probes dead founder, self-forms -> 250; founder rejoin -> 240.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
All 6 failover drills pass on the docker-dev three-mesh rig:
- D1 MAIN graceful manual-failover via the AdminUI "Trigger failover" button —
Primary leaves, peer→250, restarts and rejoins with no split.
- D2 SITE-A auto-down failover, both directions; D3 SITE-B crash-the-oldest.
- D4 auto-down 1-vs-1 survival — every survivor stayed Up (closes the gate
deferred since Phase 0a).
- D5 self-first cold-start-alone — a node boots with its partner down and forms
its own mesh (gate b).
- D6 recovery — restarted nodes rejoin as Secondary (240).
Ships a one-page co-located operator runbook (OtOpcUa + ScadaBridge on shared
site VMs). Manual-failover button confirmed MAIN-only by construction; site
pairs (driver-only, no UI) fail over via auto-down. Carried Phase-6 finding —
simultaneous cold-start of both pair VMs can split — mitigated operationally in
the runbook (staggered start); a product-level join guard is a candidate
follow-up, not shipped.
The per-cluster mesh program (Phases 0a–7) is COMPLETE.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Records the full live-gate run against master on the docker-dev three-mesh rig.
All 8 legs pass: three isolated 2-node meshes, one Primary per pair (250/240),
deploy sealed acks=6 over per-cluster ClusterClient, telemetry gRPC :4056 up +
reconnect, reconciler own-cluster-scoped, secrets pair-local, split validator
refuses Dps, and per-pair failover with auto-down 1-vs-1 survival (also closes the
deferred Phase 0a gate). Three defects were caught and fixed forward (3a4ed8dd,
792f28ec) — see the live-gate doc's Findings section. Task 9 marked completed.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Two docker-dev-only defects the Phase 6 live gate surfaced (both invisible to
`docker compose config`, which validates the file but never boots the app):
1. Driver-only site nodes crashed at boot with OptionsValidationException — LDAP
transport None + AllowInsecure false. Only central carried the Security__Ldap__*
block, but the site nodes serve OPC UA endpoints and validate LDAP options too, and
they override `environment` wholesale (YAML `<<:` doesn't deep-merge it). Extracted
the block to a shared &ldap-env anchor merged into every host node.
2. Site pairs split-brained at simultaneous startup (both nodes "JOINING itself",
250/250 instead of 250/240). Both are self-first seeds, so both run Akka's
FirstSeedNodeProcess; the partner didn't depend on its pair founder, so they raced
and each formed a 1-node cluster. Added an &akka-founder-healthcheck (bash /dev/tcp
probe of Akka 4053 — no nc/curl in the image) to site-a-1/site-b-1 and switched the
partners to depends_on service_healthy, so the founder forms first and the partner
JOINS it. (Carry the underlying simultaneous-cold-start property to Phase 7 — it is
inherent to symmetric self-first seeding, present for central since #459.)
Live gate record: docs/plans/2026-07-24-mesh-phase6-live-gate.md (legs 1-2 PASS —
three isolated meshes, one Primary per pair at 250/240).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Redundancy.md: KNOWN-LIMITATION (per-Akka-cluster election) marked RESOLVED;
new 'Per-cluster meshes (Phase 6)' section + pair-local secrets note; manual-
failover mesh-scope caveat removed; auto-down 1-vs-1 note updated (every mesh is
now two nodes, drill deferred to Phase 7). IManualFailoverService + ClusterRedundancy
razor caveats updated to pair-local reality. Configuration.md documents the
cluster-{ClusterId} role, per-pair self-first seeds, SplitTopologyTransportValidator,
and the intentional Dps default. Program + design status tables + CLAUDE.md marked
Phase 6 DONE.
(Task 8 subagent completed the edits but died on an API error before committing;
reviewed and committed by the controller.)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 6 splits the fleet into one mesh per Cluster, so an admin (central)
node's Cluster.State.Members shows only its own pair — it no longer sees
site members via gossip. The ClusterNodeAddressReconciler singleton compared
live membership against EVERY ClusterNode row fleet-wide, so post-split every
foreign-cluster row would log EnabledRowNotInCluster forever.
Scope the actor's two DB queries to rows whose ClusterId matches the admin
node's own (IClusterRoleInfo.ClusterId, sourced at registration). A legacy
admin node with no cluster role (null ClusterId) still reconciles the whole
fleet — it genuinely sees every member via gossip. The pure Reconcile
function and AddressMismatchKind semantics are unchanged; scoping lives
entirely in the actor's queries.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Plan + task ledger for splitting the single fleet mesh into three 2-node
meshes (central + per-site pairs), cluster-scoped roles + redundancy
singleton, per-cluster ClusterClient, own-cluster reconciler, split-topology
transport validator + flipped defaults, pair-local secrets, and the docker-dev
rig rewrite. Decisions settled with the user 2026-07-24.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Full 12-stream telemetry mesh formed in Grpc mode, AdminUI pill live (data path
proven), kill-and-reconnect recovered a node's stream in ~5s; Dps baseline dials
nothing. Surfaced+validated the ClusterNode.GrpcPort upgrade-backfill gotcha.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Deploy sealed green with 4 DB-less site nodes acking (central persists acks);
ServiceLevel held 240 on a site node with central SQL stopped (survive-alone);
restarted site node booted last-known-good from the LocalDb pointer, no ConfigDb
read; alarm_condition_state table live + replicated. Full record in
2026-07-23-mesh-phase4-live-gate.md. Task 9 (drop dead table) deferred.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Task 4+6 code review re-surfaced the OpcUaPublishActor raw-sink wipe on a
null-factory node (the exact #485/#486 defect Task 3 fixes) — Task 3 must land
before any node flips to FetchAndCache. Ordering already enforces this
(3 → 7 → 10 → 11).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Nullable ConfigDb factory; UpsertNodeDeploymentState no-ops when absent
(central persists acks from the ApplyAck); scripted-alarm condition state
served from the LocalDb store instead of the ConfigDb-backed Ef store.
Combines Phase-4 Tasks 4 + 6. Removes the interim dbFactory! cast.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Task 2 code review surfaced two unguarded ConfigDb sites reachable on a
driver-only node once Tasks 0+1 landed (SpawnScriptedAlarmHost's Ef store,
unconditional UpsertNodeDeploymentState). Task 4 brief now names them + the
dbFactory! removal; Task 6 re-homes the alarm store to LocalDb.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 4 of the per-cluster mesh program. Removes the four remaining driver-side
ConfigDb consumers + the registration itself so a driver-only node boots with no
ConfigDb connection string:
- ConfigDb registered iff hasAdmin (Program.cs); driver-only requires none
- driver-only ⇒ FetchAndCache mandatory (validator)
- DbHealthProbeActor not spawned on driver-only; DbReachable=true constant
(client-visible ServiceLevel change — a healthy DB-less node publishes 240/250
with central down, per user decision 2026-07-23)
- DriverHostActor: nullable factory, drop redundant SQL ack-writes (central's
PersistNodeAck is the ack system of record)
- EfAlarmConditionStateStore → LocalDbAlarmConditionStateStore (new replicated
alarm_condition_state table)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 3 Task 9. docker-dev rig: central-1/2 carry ConfigServe (:4055 h2c + committed
dev key) and stay Direct; the four site nodes carry ConfigSource (both central
endpoints + matching key) defaulting to Direct. Flip only the site nodes with
OTOPCUA_CONFIG_MODE=FetchAndCache at 'docker compose up' — central keeps SQL and models
the target topology. Binding :4055 activates the dedicated-h2c Kestrel takeover on
central (re-binds :9000 too). Docs: new docs/Configuration.md ConfigSource/ConfigServe
section; docs/Redundancy.md 'config bytes travel out-of-band' note; the program plan
Phase 3 section + tracking row flipped to CODE-COMPLETE; a CLAUDE.md 'Config source'
section.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Eleven tasks. User decisions baked in: gRPC fetch RPC (not token-gated HTTP),
shared node bearer key (not a per-deployment token, so no migration and
DispatchDeployment is unchanged), a ConfigSource:Mode dark switch (Direct
default), both pair nodes fetch (no Primary gating). Header carries the five
recon facts and the Phase 3/4 boundary (config READS only; the NodeDeploymentState
write, DbHealthProbe and EfAlarmConditionStateStore stay for Phase 4).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Flipped the docker-dev rig end-to-end to MeshTransport:Mode=ClusterClient
(OTOPCUA_MESH_MODE, no rebuild) and ran the gate. PASSED for the transport:
deploy seals on ClusterClient, a stopped node fails the deploy naming it, the
MaintenanceMode hatch shrinks the DB-sourced contact set live (6->5->6 on both
centrals independently), and the frame-size canary stayed silent.
Four findings, recorded in 2026-07-22-mesh-phase2-live-gate.md:
A. A 10-30s post-restart delivery window exists, but the DPS control fails
identically -- it is a cluster-convergence property, not a Phase 2
regression. Running the control is the only reason this is a note and not
a false attribution.
B. buffer-size=0's stated rationale was OVERSTATED. A node applies a TimedOut
deployment anyway -- live-observed at +44s -- via DriverHostActor.Bootstrap
replaying the orphan Applying row on restart. Zero buffering removes the
silent, timer-driven replay, not the boot-time DB one. Corrected in the
plan decisions table, Configuration.md, and CLAUDE.md; the decision stands,
the justification narrowed.
C. Stopping BOTH centrals strands the non-seed nodes at the Akka membership
layer (frozen mid-Exiting, not Unreachable, so auto-down does not help).
Transport-independent; a Phase 7 downing/rejoin drill item.
D. The AdminUI driver Restart/Reconnect buttons (DriverStatusPanel) are
rendered by no page -- orphaned since v3 Batch 2 retired /clusters/{id}/
drivers. So gate steps 6/7 could not run; the driver-control node-side leg
is proven live only because DispatchDeployment shares its EventStream path.
Step 8's planned log-signal does not exist (HandleApplyAck is silent on
success); proven instead by both centrals independently logging per-instance
ClusterClient creation, which a singleton cannot.
Also adds a repo-root .dockerignore: docker-dev/Dockerfile does COPY . ., which
was pulling .claude/worktrees (19GB of agent checkouts with their own bin/obj)
into the build context and exhausting the Docker VM disk mid-COPY.
Rig restored to Mode=Dps, six members, no leftover maintenance flags.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Joseph Doherty