The issue reports a missing feature. The cause is a stale projection, and the
consequence was worse than the gap described.
`ScriptTagCatalog` still emitted `Tag.TagConfig.FullName` — the pre-v3 driver wire
address. Since v3 the mux key is the **RawPath**: `AddressSpaceComposer` harvests the
`ctx.GetTag("…")` literals into `DependencyRefs`, those become `VirtualTagActor`
dependency refs registered with `DependencyMuxActor`, whose `_byRef` is keyed Ordinal
by `AttributeValuePublished.FullReference` — and a driver's wire ref IS the RawPath
(`DriverHostActor._nodeIdByDriverRef` is keyed `(DriverInstanceId, RawPath)`). The
`{{equip}}/<RefName>` form already substitutes to RawPaths at both compose seams.
So the catalog was wrong in BOTH directions, not merely incomplete: completion
offered paths that could never resolve at runtime, while a CORRECT absolute RawPath
got no completion and hovered as "not a known configured tag path". The file's own
comment admitted the raw/UNS catalog was left for "Batch 2/3" and it never came.
Now projected through the shared `RawPathResolver` — the same byte-parity authority
`DraftValidator` and `AddressSpaceComposer` use — so a suggested path is identical to
the one the deploy gate and the runtime compute. A tag with a broken ancestry chain
resolves to null and is omitted: the runtime could not route it either, so offering
it would be a lie. Hover additionally gains the owning `DriverInstanceId`, which the
topology now makes available (it was always null before).
Editor-accepts <=> publish-accepts is preserved: no diagnostic keys off this catalog
(`OTSCRIPT_EQUIPREF` only marks `{{equip}}` paths), so this changes completion and
hover only — it cannot newly reject a script.
The existing tests pinned the v2 contract and were re-authored: a RawPath only exists
if the RawFolder -> DriverInstance -> Device -> TagGroup -> Tag chain does, so the
seed now builds one. Added coverage for group-nested paths, a driver at the cluster
root (no folder segment), a broken chain being omitted rather than guessed, and the
pre-v3 FullName no longer resolving.
Live-gated on docker-dev (rig rebuilt, both central nodes):
- completion, empty literal -> every RawPath incl. group-nested
`opcua1/plc/OpcPlc/Telemetry/Basic/AlternatingBoolean`
- completion, prefix `pymod` -> `pymodbus/plc/HR200X`, `pymodbus/plc/ImportedTag`
- hover `pymodbus/plc/HR200X` -> "Tag · Type UInt16 · Driver MAIN-modbus"
- hover an unknown path -> "Not a known configured tag path"
AdminUI 742/742; script-analysis 55/55; solution builds 0 errors.
Wave 1 of the driver-expansion program. The branch was complete and live-gated but
had never been merged; it sat 15 commits ahead and 50 behind master.
Purely additive behind the existing IModbusTransport seam — the application protocol
(function codes, codecs, planner, coalescing, write path, probe, materialisation,
HistoryRead) is identical across TCP and RTU; only the wire framing differs
([addr][PDU][CRC-16], no MBAP, no TxId). Zero changes to codecs/planner/health.
Selected by a new Transport config field (ModbusTransportMode: Tcp | RtuOverTcp),
routed through ModbusTransportFactory (throws on unknown mode), with the string-enum
guard on options/DTO/factory and a Transport selector on the AdminUI Modbus form.
ModbusSocketLifecycle was extracted from ModbusTcpTransport (behaviour-preserving)
and is composed by both transports.
Descoped by design: direct-serial (System.IO.Ports) and Modbus ASCII — RTU buses are
reached exclusively via a serial->Ethernet gateway. No per-tag config changes; the
existing per-tag UnitId already makes a multi-drop bus work.
Re-validated against current master at merge time (the branch's own gate predates 50
commits of master): no overlapping files, clean merge, solution builds 0 errors, and
Modbus 344/344, Modbus.Addressing 161/161, AdminUI 740/740, Runtime 507/507,
Configuration 131/131 all pass. Confirmed the Transport selector is still reachable
through master's current authoring path (DriverConfigModal -> ModbusDriverForm), so
the branch's live /run gate result still applies.
Both defects are PRE-EXISTING (present on master, unrelated to #499) and were found
live-verifying the #499 guard, which lives on this page and was unreachable without
them fixed.
1. **HTTP 500 on load.** `FormModel.ServiceLevelBase` was `byte` to match the
entity, and `InputNumber<T>`'s static constructor throws outright for
`System.Byte` — "The type 'System.Byte' is not a supported numeric type". That
escapes `BuildRenderTree` unhandled, which in Blazor takes the WHOLE page down,
not one field. `/clusters/{id}/nodes/{nodeId}` and `.../nodes/new` have never
rendered. Now `int` with the same `[Range(0, 255)]` and a cast on the way to the
entity, so the stored type is unchanged. (Same failure mode as #504: an
unhandled throw inside the render tree is a page-kill.)
2. **Save did nothing, silently.** `NodeId` was validated against
`^[A-Za-z0-9_-]+$`, which forbids the colon — but every NodeId in this system is
`host:port` (`ClusterRoleInfo` and `ConfigPublishCoordinator` derive it as
`member.Address.Host:Port`, the seed writes `central-1:4053`, and
`NodeDeploymentState.NodeId` is FK-bound to it). So every existing node failed
validation, `OnValidSubmit` never ran, and the button was inert. Pattern now
allows `:` and `.` (hostnames may be dotted).
3. **Added the missing `<ValidationSummary/>`.** The form had a
`DataAnnotationsValidator` but nothing rendering its output, so a validation
failure produced no feedback at all — which is precisely how (2) stayed
invisible. This is the change that stops the next such defect being silent.
Live-verified on docker-dev (both central nodes rebuilt, since :9200 round-robins
the pair): page 500 → 200 on both routes, and a real edit now persists.
Option A from the issue — the cheap consistency check, no interface change.
The desired ref set was re-applied on a Connected TRANSITION (`ResubscribeDesired`)
and on a deploy's `SetDesiredSubscriptions`, and nowhere else. A subscription lost
while the driver STAYED Connected — a subscribe that failed and was never retried,
a handle dropped down an error path — left the actor subscribed to nothing while
still reporting Healthy, indefinitely.
`ReconcileSubscription` now rides the existing 30 s `health-poll` tick: while
Connected, a driver holding a desired set but no live handle is re-subscribed.
Gated on `ISubscribable`. A driver that cannot subscribe at all can still be handed
a desired set, and without the guard it would be told `Subscribe` every tick and
fail every tick, forever.
This is a state-machine consistency check, NOT a probe — it can only see that this
actor holds no handle, never that a handle is present but dead server-side, because
`ISubscriptionHandle` is opaque. That is option B, and it stays deferred until a
real occurrence shows the handle outliving the subscription; it would need a
liveness member implemented across all eight drivers with different subscription
mechanics.
A redundant `Subscribe` is possible (a tick queued ahead of an already-queued one
observes the same no-handle state) and is harmless: `HandleSubscribeAsync` is a
`ReceiveAsync`, so no tick is processed mid-subscribe, and it drops any prior
subscription before establishing the new one. Suppressing it would need a
pending-subscribe flag threaded through every self-tell site — more state than the
race is worth.
`healthPollInterval` becomes an optional Props parameter, mirroring the existing
`rediscoverInterval` seam, so a test can drive the reconcile without waiting 30 s.
Three tests. The reconcile itself is proven by a stub whose FIRST subscribe throws:
nothing else would ever retry it, so a second attempt can only come from the
reconcile — disabling `ReconcileSubscription` turns it red. The two guard tests are
absence assertions and are bounded by a positive control rather than a sleep: a
counting health publisher makes ticks PROCESSED observable, so "still one subscribe"
means the reconcile ran and declined, not that the timer had not fired.
Explicitly not deploy-time drift (#486) — re-asserting an already-correct desired
set would not have helped there.
`ClusterNode.DriverConfigOverridesJson` is the third surface a Sql driver's config
is persisted on, and the only one nothing gated. It is a map keyed by
`DriverInstanceId` merged onto the cluster-level `DriverConfig`, so a literal
`connectionString` pasted there leaks exactly as one pasted into the driver — the
leak #498 closed for `DriverInstance.DriverConfig` and `Device.DeviceConfig`.
Gated at the SAVE, not at the deploy — a deliberate departure from both options the
issue offered:
- `DraftSnapshot` carries no `ClusterNode` rows, and adding them would widen the
snapshot and every builder of one for a single rule about a value that never
enters the artifact.
- More to the point, a deploy gate is the wrong instrument here. Node overrides are
not in the artifact, so blocking a deploy would not stop the credential being
stored — it is already in the database and replicated by then. Refusing the save
is the only point where "refuse to store it" is literally true, which is #498's
own framing: discarding a secret on read is not the same as refusing to store it.
The check moves into a shared `SqlCredentialGuard` so the two enforcement points
cannot drift; `DraftValidator` now calls it instead of its own private helper.
Kept deliberately narrow — the `Sql` driver's `connectionString` only, and only for
instance ids that ARE Sql drivers. The issue asked whether to generalise to
credential-shaped keys across every driver type; not doing that, for the same
reason #498 did not: a broader sweep would start refusing configs that are
legitimate today for drivers which never made an indirect-credential guarantee,
which is a regression rather than defence in depth. Widen per driver, as each gains
its own contract.
The error names the offending driver instance(s) and NEVER the value — it reaches
the AdminUI and the audit trail.
14 new cases cover both halves, including the case variants (System.Text.Json binds
`ConnectionString` to `connectionString`, so a case variant is the same key, not a
bypass), non-Sql ids being ignored, and every blank/malformed/non-object shape.
`Scripts.razor` rendered a hash prefix as `@s.SourceHash[..12]…`. `SourceHash`
is a plain nvarchar(64) with no length floor, so any value shorter than 12
characters threw ArgumentOutOfRangeException out of BuildRenderTree — unhandled
in Blazor, which takes the WHOLE page to an HTTP 500, not just the offending
row. Live-reproduced on docker-dev, where two Script rows carry `h1` and
`h-abs-hr200`: /scripts was a hard 500.
New `Components/Shared/DisplayText.Abbreviate(string?, int)`:
- null/blank renders the Admin UI's em-dash placeholder rather than throwing;
- a value already within budget renders verbatim with NO ellipsis, so the
display never implies text that isn't there (the old markup appended "…"
unconditionally, outside the slice — `h1` would have shown as `h1…`);
- a maxLength < 1 is clamped, so a display bug cannot become a page crash.
Applied to every unguarded range-operator slice over a DB-sourced string found
by sweeping the AdminUI for `[..N]`:
Scripts.razor SourceHash (the reported crash)
Certificates.razor Thumbprint (read off the on-disk store)
Deployments.razor x2 RevisionHash
Clusters/ClusterOverview.razor RevisionHash
The other `[..N]` hits are safe and untouched: Guid.ToString("N") slices
(always 32 chars) and the `Length > 60 ? [..60]` ternaries that self-guard.
NOTE — the issue text blamed the docker-dev seed; that was wrong and is
corrected on #504. `seed-clusters.sql` inserts only ServerCluster, ClusterNode
and LdapGroupRoleMapping. The short-hash rows came from earlier live-gate
authoring. A freshly seeded rig is fine; the trigger is any Script row that did
not go through ScriptEdit / UnsTreeService.HashSource — REST-API authored,
hand-inserted, or migrated in.
Tests: 14 new in DisplayTextTests, covering the short/null/blank/clamped cases
and a never-throws sweep over every value x budget combination. AdminUI suite
739/739.
Live-verified on docker-dev (AdminUI has no bUnit): /scripts 500 -> 200 across
both central nodes, rendering `hash=h1` and `hash=h-abs-hr200` in full;
/deployments still truncates 64-char hashes at 12 with the ellipsis;
/certificates and /clusters/MAIN unchanged.
A deploy that triggers a FULL address-space rebuild clears `_alarmConditions`
and re-materialises every condition node fresh — inactive, acked, confirmed,
unshelved. The engine reloads its persisted state and re-derives Active from the
predicate, so it ends up correct; but there is no transition to report, so
`LoadAsync` yields `EmissionKind.None` and `OnEngineEmission` drops it. Nothing
writes the node.
Live-reproduced on the docker-dev rig against the pre-fix image: an alarm that
had fired and cleared but was still UNACKNOWLEDGED came back from a rebuild
reporting Acknowledged with Retain=false — so it disappeared from
ConditionRefresh entirely, while the engine still held Unacknowledged. An
operator's outstanding alarm silently vanishes from the alarm list on deploy.
`ScriptedAlarmHostActor.ReassertConditionNodes` closes it. After each load it
reads the new `ScriptedAlarmEngine.GetProjections()` and sends one
`AlarmStateUpdate` per alarm NOT in the Part 9 no-event position.
Load-bearing properties:
- Node-only. It sends `AlarmStateUpdate` and nothing else — never the `alerts`
topic, never the telemetry hub. `alerts` is the historization path, so a row
per alarm per deploy would append a duplicate historian/AVEVA record every
time anyone deploys. This is why it reads a projection rather than re-emitting:
`ScriptedAlarmProjection` carries no `EmissionKind`, so there is nothing for
the emission machinery — or a future refactor — to mistake for a transition.
The issue's sketch said `GetState(alarmId)` + `LoadedAlarmIds`; that alone
would have clobbered the node's severity and message, so the projection also
carries the definition's severity, the resolved message template, and the
last-observed worst-input quality.
- Only non-normal alarms. One in the no-event position already matches what the
materialise built, so a never-fired alarm stays byte-identical to a deploy
without this pass (including its Message, which the materialise seeds with the
display name), and an all-normal deploy writes nothing at all.
- Timestamp is the condition's own `LastTransitionUtc`, not the deploy instant.
- No duplicate Part 9 events: `WriteAlarmCondition`'s delta-gate compares against
the node's CURRENT state, so a re-assert onto a freshly materialised node is a
genuine delta (fires once — correct, the rebuild reset what clients see) while
one onto a surgically-preserved node is suppressed.
Tests: 6 new host tests + 6 new engine tests. Falsifiability checked by
disabling the call — 4 of the 6 host tests go red; the other 2 are absence
guards against an over-broad fix and pass either way by design.
Live gate (docker-dev, central-2, pre-fix image vs. rebuilt image):
- pre-fix: condition absent from ConditionRefresh after a rebuild;
- post-fix: present, Unacknowledged, Retain=True, stamped with its own
LastTransitionUtc rather than the restart instant;
- /alerts stayed empty across two re-asserts, with the panel proven live by a
real ACTIVATED transition immediately afterwards — no duplicate history.
Task 26 — the P2 milestone gate. Ran the live /run verification on an isolated
docker-dev rig (project otopcua-mqtt, ports 9210/4850-4851/14350, image built
from this branch) against the real Mosquitto TLS+auth broker and the C#
Sparkplug edge-node simulator on 10.100.0.35, and recorded the result.
The gate found three defects. Two are the same defect class the P1 gate found
twice — a hand-maintained AdminUI surface left behind by a driver-side feature —
and the third is a pre-existing cross-driver bug that only a Sparkplug flow
could surface.
1. CRITICAL, FIXED — MqttDriverForm still shipped its P1 Sparkplug PLACEHOLDER.
Switching Mode to SparkplugB rendered a "not available yet" notice and NO
Group ID field, so with Sparkplug ingest fully shipped there was still no
way to author a Sparkplug driver from the AdminUI at all. Sparkplug.GroupId
is the driver's entire subscription filter (spBv1.0/{GroupId}/#): blank ⇒
connected, Healthy, ingesting nothing. Now authors all five Sparkplug keys,
MERGES over the existing sub-object rather than replacing it (so a key a
newer driver adds inside it survives an older AdminUI), leaves it untouched
in Plain mode, and validates the group id as the topic segment it is.
Pinned by 8 MqttDriverFormModelTests cases, verified falsifiable — 8 RED
with the fix stubbed out.
2. PRE-EXISTING + CROSS-DRIVER, FIXED — every node label in the shared
DriverBrowseTree was an <a href="#">. In a Blazor Web App, blazor.web.js's
enhanced-navigation click interceptor resolves a bare "#" against
<base href="/">, so clicking ANY browse-tree node label navigated the whole
AdminUI to "/", tore down the circuit, and destroyed the hosting modal —
losing the browse session AND the tag selection. @onclick:preventDefault
does not help: it suppresses the browser's default action, not Blazor's own
interceptor. Labels are now <button type="button">. Dates to the component's
introduction (2026-05-28) and affects EVERY driver's picker; it surfaced
only now because choosing a rebirth scope is the first flow that requires
clicking a label rather than the ▶ toggle (always a button, always fine).
3. FIXED (mitigation) — the browse tree rendered exactly once, at open, and
never again, while OpenAsync returns as soon as the SUBSCRIBE is granted.
On Sparkplug that means it is almost always empty forever, because births
are never retained. Added a Refresh button that re-reads the root against
the SAME session (nothing reconnects, nothing observed is lost, the tag
selection is kept, only the armed rebirth scope is cleared).
Live gate result (all against the real broker + simulator, group OtOpcUaSim):
- driver authored end-to-end through the fixed MqttDriverForm; blob is
Mode:SparkplugB + Sparkplug.GroupId:OtOpcUaSim with enums as names
- browse: BROWSER OPEN chip, Sparkplug-only rebirth panel, tree renders
Group → EdgeNode → [Device] → Metric with the device folder Filler1 and
node-level metric leaves SIDE BY SIDE, and "Node Control/Rebirth" as ONE
leaf (the metric-name-is-not-a-topic-segment rule holding)
- rebirth: group scope names the group + the 32-node cap, Cancel published
NOTHING (simulator log unchanged); edge-node scope → "1 command published"
and the simulator logged "rebirth NCMD #1 received; republishing metadata";
group scope → "2 commands published", both edge nodes re-birthed; a device
and a metric both resolve UP to "edge node EdgeA"; the panel does NOT render
for a Plain MQTT device
- browse-commit wrote bindable tuples with no topic/address key —
{"groupId","edgeNodeId","metricName","deviceId"} for the device metric and
deviceId ABSENT (not blank) for the node-level one, datatypes inherited from
the birth (Int64 / Float)
- deployed (Sealed), and both nodes served live changing values through OPC UA
at the 2 s cadence — Good 0x00000000, no BadNodeIdUnknown
- death→STALE: stopping the simulator drove both nodes to 0x80000000 with an
empty value ~2 s later, and restarting it recovered them to Good on the new
birth (FillCount back to its birth-declared 1000, then counting)
- rediscovery fired exactly TWICE (once per authored scope: OtOpcUaSim/EdgeA
7 metrics, OtOpcUaSim/EdgeA/Filler1 3 metrics) and was then gated across
many subsequent births/rebirths — the anti-storm change gate working, and
EdgeB correctly filtered out as an unauthored scope
- tag editor: opens in SparkplugB (mode inference on reopen) with all four
descriptor fields populated; blank group → "A Sparkplug group ID is
required."; "Plant/1" rejected; "Node Control/Rebirth" ACCEPTED and saved
with the slash whole; dataType override persists and its key is ABSENT again
after selecting "(from birth certificate)"
Two things are NOT verified, and are recorded rather than claimed:
- Rediscovery is INERT in v3 and this is a platform gap, not an MQTT one:
nothing subscribes to IRediscoverable.OnRediscoveryNeeded and
DriverHostActor.HandleDiscoveredNodes hard-returns. A DBIRTH introducing a
metric does NOT change the OPC UA tree; redeploy. Verified from the driver's
log line only, exactly as far as it can be verified.
- A metric name containing "/" cannot be BROWSE-COMMITTED: the derived raw tag
Name is a RawPath segment and may not contain "/", so the spec-mandatory
"Node Control/Rebirth" is refused ("Row 3: Name must not contain '/'"). The
refusal is loud and all-or-nothing, never a silently mis-bound tag, and
Manual entry is the working path. Fixing it needs a name-sanitisation policy
with a collision answer, so it was recorded rather than guessed at.
Also left open + documented: an empty browse tree cannot arm a rebirth (the
scope comes from a tree click, and the session side already accepts a bare
{group}/{edgeNode} — only a UI path is missing); _canRebirth is captured at
browse-open; the tag editor injects the Plain-only payloadFormat key into a
Sparkplug blob (cosmetic — the factory routes on the tuple).
Suites: offline 1528 passed / 0 failed (581 Driver.Mqtt + 809 AdminUI + 138
Core.Abstractions); live 15/15 against the broker + simulator.
Docs: docs/drivers/Mqtt.md brought fully up to date for P2 (Sparkplug config
sub-object, both tag shapes, the ingest state machine, rediscovery, browse +
rebirth + Refresh, a Known-gaps table); Mqtt-Test-Fixture.md gains the simulator
and drops its "Sparkplug has no fixture" claims; infra/README.md §3 gains the
simulator row; CLAUDE.md gains the simulator endpoint facts; the tracking doc
marks MQTT/Sparkplug COMPLETE. docker-dev/docker-compose.mqtt.yml is the
isolated-rig overlay the gate ran on.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Two gaps the Task 23/24 review found, both blocking Task 26's live gate.
Gap 1 — browse-commit produced a silently dead MQTT tag. RawBrowseCommitMapper
had no `Mqtt` case, so a committed leaf fell through to the generic
`{"address": …}` key. Neither MqttTagDefinitionFactory entry point reads
`address`: the tag deployed clean and reported BadNodeIdUnknown forever, with
no signal at commit time. Predates Task 23 (Plain was affected too), but Task 23
built the Sparkplug metric tree precisely so an operator could browse and commit
a binding.
The address is a DESCRIPTOR, not a reference string, and it cannot be recovered
from the browse node id: `{group}/{node}[/{device}]::{metric}` where a metric
name legitimately contains `/` (`Node Control/Rebirth`) — the ambiguity
MetricSeparator's remarks already name. So the session STATES it, via a new
`AttributeInfo.AddressFields` seam, and the mapper reads it. Which keys are
emitted is also how the mapper learns Plain vs Sparkplug — the driver type
reaching it is just `Mqtt`, and the mode lives on the driver config. Key names
are single-sourced in the new `MqttTagConfigKeys` (producer, mapper, factory),
with the literals pinned by a test so a symmetric rename cannot silently unbind
already-persisted blobs. A leaf with no stated address is refused at commit in
words rather than committed dead.
Gap 2 — RequestRebirthAsync had no UI. Task 23 shipped it backend-only; Task 26
step 1 assumes the button, and Task 25's runbook notes the picker tree stays
empty until a birth lands, so it is the only way to fill it on demand. Added to
the /raw browse modal: scope = the clicked tree node (device/metric resolve up to
their edge node, group fans out and is refused whole past 32), an explicit
two-click confirm naming the resolved scope and its consequence, the outcome or
the refusal shown rather than swallowed. Offered only for a Sparkplug session
(new `IRebirthCapableBrowseSession.RebirthAvailable` — one session class serves
both modes, so a type test alone would draw a button that can only throw) and
only to a DriverOperator; the server-side gate remains the boundary.
Tests: MQTT 545 → 581, AdminUI 781 → 800. The round-trip tests feed the emitted
blob to the REAL factory and were falsified by mutating the emitted key name.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Unseals the Sparkplug branch of the MQTT address-picker browser that Task 10
deliberately sealed shut, and adds the first (and only) sanctioned publish on a
browse session.
- MqttBrowseSession serves Group -> EdgeNode -> [Device] -> Metric in Sparkplug
mode, decoded from observed NBIRTH/DBIRTH certificates (the only Sparkplug
messages that name their metrics). Node-level metrics hang directly under
their edge node -- no synthesised device folder, because the authored address
tuple genuinely has deviceId = null for them. Metric node ids use a '::'
separator: a metric name may contain '/', so slash-joining would make a
node-level metric indistinguishable from a device folder.
- AttributesAsync is purely birth-derived in Sparkplug mode. The plain path's
UTF-8 inference / payload-snippet machinery does not run and no payload bytes
are retained: a Sparkplug body is protobuf and would be reported as
"(N bytes, binary)" for every metric in the plant. A datatype ToDriverDataType
cannot map is reported as the Sparkplug type name, never coerced.
- RequestRebirthAsync(scope) is the one publishing member, reached only by an
explicit operator action. It routes through the counted PublishAsync
chokepoint via a private RebirthTransport adapter, so PublishCountForTest
still proves that OpenAsync/RootAsync/ExpandAsync/AttributesAsync/Observe/
DisposeAsync publish nothing -- now in both modes. A device or metric scope
resolves up to its owning edge node (NCMD is node-addressed); group scope
enumerates observed edge nodes and is refused whole past
MaxGroupRebirthNodes = 32.
- BrowserSessionService.RequestRebirthAsync gates on the same DriverOperator
policy that gates the picker, evaluated server-side where the action happens
rather than only at render time, fails closed, and Info-logs the scope and the
published count. Exposed through the new IRebirthCapableBrowseSession contract
so "does this session write?" stays a type question.
- ToBrowseOptions' Last-Will landmine re-verified: MqttDriverOptions still
carries nothing will-shaped (an NDEATH is a broker-published will and would be
invisible to PublishCountForTest), now pinned by a reflection guard.
Falsifiability: injecting a publish into RootAsync reddens the Sparkplug and
plain passivity tests; double-publishing per target reddens all four
one-NCMD-per-node assertions. 572/572 MQTT tests, 24/24 AdminUI browsing tests.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Task 24 (P2): replaces the P1 Sparkplug Validate() stub (always null) and the
"not available yet" editor placeholder with real groupId/edgeNodeId/deviceId/
metricName authoring, mirroring MqttTagDefinitionFactory.FromSparkplugTagConfig
in both directions -- required ids + optional deviceId, dataType/qos read
strictly but only when present (dataType is optional: the birth certificate
declares it). One rule is deliberately stricter than the factory: group/edge-
node/device ids reject '/', '+', '#' (topic-segment characters a decoded
incoming id can never carry, so an authored one that does can never bind);
metricName is exempt since Sparkplug's own canonical names use '/' (e.g.
"Node Control/Rebirth"). No FullName key is written -- TagConfig carries no
identity under v3, and the plan's snippet asking for one was corrected per the
brief. A SparkplugB tag now survives save/reopen because its descriptor keys
re-infer Mode on load; there was never a 'mode' key to persist.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The P1 live gate had to insert MQTT driver config directly via SQL: DriverConfigModal and
DeviceModal both fell through to "No typed config form for driver type Mqtt" with no raw-JSON
fallback, so broker host/port/TLS/credentials were unauthorable from the AdminUI. Task 12 built
the *tag* editor; these are the driver + device surfaces.
MqttDriverForm authors the whole MqttDriverOptions connection surface (host/port/clientId,
TLS + CA pin, credentials, protocol version/clean-session/keep-alive, connect timeout, reconnect
backoffs, mode, maxPayloadBytes, and the Plain sub-object). The Sparkplug sub-object stays P2 —
a marked placeholder mirroring MqttTagConfigEditor's stub, with any existing sparkplug keys
preserved untouched.
The connection is authored on the DRIVER, not the device — unlike Modbus/S7/OpcUaClient and
contrary to what docs/drivers/Mqtt.md said. DriverDeviceConfigMerger merges a device's keys up
only when the driver has exactly ONE device, so a device-authored broker connection vanishes
silently the moment a second device is added. MqttDeviceForm is therefore informational (the
GalaxyDeviceForm shape) and round-trips DeviceConfig verbatim; it flags legacy connection keys
left on a device by a pre-form deployment, because those still win via merge-up.
Serialization goes through the ONE shared MqttJson.Options instance (Task 9's decision), never a
fifth per-form copy — pinned by an assertion that an ordinal-only reader CANNOT bind the emitted
blob, and by extending the fleet-wide DriverPageJsonConverterTests guard to resolve a form's
serializer from an explicit external-instance registry when it has no _jsonOpts field. Dropping
the converter from MqttJson.Options reddens 3 tests and leaves the symmetric round-trip green —
the Task 9 lesson, reproduced.
RawTags is stripped from the emitted blob (the deploy artifact owns it) and unknown top-level
keys survive a load->save. Validation mirrors the driver's own [Range] bounds inline, and
ToOptions() additionally clamps, so an ignored error still cannot persist a
connectTimeoutSeconds:0 driver-brick. A non-blank clientId raises the P1 live-gate warning
inline (fixed ids make redundant pair nodes evict each other while both report Healthy).
AdminUI 761/761 green (was 730), TWAE-clean; Driver.Mqtt 266/266 green.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The read-only Sql driver's runtime factory/probe and tag-side editors were
wired, but the AdminUI could not author the DRIVER: the /raw "New driver" type
list, the DriverConfigModal switch, and the legacy identity dropdown all omitted
Sql, and no SqlDriverForm existed. Live-driving the AdminUI surfaced this.
- Add SqlDriverForm.razor over the SqlDriverConfigDto shape: Provider
(SqlServer-only in v1), required connectionStringRef (an env-var NAME, not a
connection string — label is explicit), optional poll/operation/command
timeouts, maxConcurrentGroups, nullIsBad. Enums serialize by NAME via
JsonStringEnumConverter; a connection-string literal can never be authored
(no such field is collected); rawTags (composer-owned) + allowWrites (inert,
read-only v1) are never emitted.
- Wire Sql into DriverConfigModal switch, RawDriverTypeDialog type list, and
DriverIdentitySection legacy dropdown.
- Add SqlDriverFormContractTests: the exact JSON the form emits constructs a
driver through the real factory, missing connectionStringRef is rejected, and
provider round-trips as the "SqlServer" name (never an ordinal).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Task 14 — the P1 milestone gate. Ran the live /run verification on a docker-dev
rig against the real Mosquitto TLS+auth fixture, and recorded the result.
The gate did its job: it found MQTT was unauthorable in production.
RawDriverTypeDialog's driver-type list is a hand-maintained array that nobody
added MQTT to, so the /raw "New driver" picker never offered it — with every
other layer (contracts, driver, browser, factory, host registration, typed tag
editor) complete and 266 green unit tests. Nothing tied that array to
DriverTypeNames and AdminUI has no bUnit, so no offline test could see it.
Fixed, plus RawDriverTypeDialogParityTests: a reflection parity guard that
fails for ANY DriverTypeNames entry missing from the picker. Verified
falsifiable — RED with "…cannot be authored from the /raw New-driver picker,
so they are unreachable in production: Mqtt" before the one-line fix.
A second gap is left OPEN and documented, not fixed (no task ever covered it):
MqttDriverForm / MqttDeviceForm do not exist, and DriverConfigModal/DeviceModal
have no raw-JSON fallback — so an operator can create an MQTT driver but cannot
author its broker endpoint or credentials. P1 is not operator-complete until
those two razor forms land. The gate authored config via SQL to get past it.
Live gate result (isolated 2-node MAIN stack, image built from this branch,
fixture at 10.100.0.35:8883, AllowUntrustedServerCertificate rather than
pinning the CA into the rig):
- typed MQTT tag editor renders and round-trips; Json shows the JSON-path
field, Raw/Scalar hide it; wildcard topic a/+/c blocked at save; data-type
list offers Float64 and no Double; qos/retainSeed absent on "(driver
default)" and qos:2 written as a number; isHistorized/historianTagname
survive a topic-only edit; SparkplugB renders the P2 placeholder and
switching back keeps Plain values; jsonPath is guidance not a gate in all
three shapes ($ seeded only for a brand-new blank tag, blank stays absent,
clearing saves fine)
- deployed, and both editor-written blobs resolved and served changing live
values through OPC UA (22.8 / 1629, StatusCode Good) — no BadNodeIdUnknown
- the bespoke #-observation browser drove a real broker session and rendered
the fixture's actual topic tree (line1/{counter,state,temperature},
noise/chatter, retained/seed); a Modbus device's picker still correctly
reports "Browsing unavailable"
Also found live and documented (not a code change): both nodes of a redundant
pair run the driver, so a FIXED clientId makes them evict each other forever —
the broker logs "already connected, closing old connection" and both nodes
reconnect every ~2s while still reporting Healthy. Unset (the default) is
correct; confirmed 0 reconnects/60s on both nodes after removing it.
Suites: offline 1134 passed / 0 failed (266 Driver.Mqtt + 730 AdminUI incl. the
3 new guards + 138 Core.Abstractions); live 7/7 against the fixture.
Docs: new docs/drivers/Mqtt.md + Mqtt-Test-Fixture.md (the per-driver + fixture
convention every other driver follows), MQTT rows in docs/drivers/README.md,
TestConnectProbes.md, root README.md, and infra/README.md §3.
CLAUDE.md drift corrected while adding the MQTT endpoints:
- the "every fixture carries project: lmxopcua" claim was false — only the
MQTT fixture does; the filter returns one stack, not the fleet
- lmxopcua-fix.ps1 is Windows-VM-only; on macOS drive the host over ssh/rsync
(and the docker-dev rig itself runs locally under OrbStack)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Review follow-up. Dropped the hard Validate rejection of a blank jsonPath under
a Json payload: the runtime defaults it to the document root, which is the real
"publisher puts a bare JSON scalar on the topic" case, so rejecting it made the
editor refuse what the driver accepts — inverting "authoring surface accepts <=>
publish accepts" — and blocked whole CSV-import batches, since this validator
also gates RawManualTagEntryModal's review grid.
Replaced with guidance: an UNAUTHORED tag (no topic AND no jsonPath) seeds the
field with "$", so a fresh tag emits an explicit "jsonPath":"$" while an
existing path-less tag keeps the key ABSENT and the driver defaults it. The
wildcard-topic rule stays strict — a wildcard has no legitimate single-Tag
interpretation.
Also: omit a blank "topic" rather than writing "topic":"", and record why the
_lastConfigJson guard diverges from the Modbus template — the landmine is a
DERIVED, NON-PERSISTED UI FIELD INFERRED FROM BLOB CONTENT (Mode), which Task
24's Sparkplug field group will be tempted to add more of. Named the host-side
dependency (RawTagModal only mutates through this callback) that keeps the
staleness trade benign today.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Task 9. Adds MqttDriverFactoryExtensions (DriverTypeName = DriverTypeNames.Mqtt,
direct deserialization of MqttDriverOptions — no intermediate DTO, mirroring
OpcUaClientDriverFactoryExtensions), the DriverTypeNames.Mqtt constant, and both
host wiring sites: the factory in DriverFactoryBootstrap.Register and the probe in
AddOtOpcUaDriverProbes (the admin-node path Program.cs calls in its hasAdmin
block — a probe wired only on driver nodes makes Test-connect silently dead).
Carried-forward items:
- Converges every MQTT config-parsing seam onto ONE JsonSerializerOptions —
MqttJson.Options, in .Contracts alongside MqttDriverOptions. It replaces the
probe's internal JsonOpts and the browser's separate private copy; the factory,
the probe, MqttDriver.ParseOptions and MqttDriverBrowser now all parse through
it. .Contracts is the only assembly all four consumers reference, and the
browser's reference to the runtime .Driver project is a documented layering
exception scheduled for removal — anchoring the options there would resurrect
the duplicate the day it goes away.
- Replaces the "Mqtt" literals in MqttDriverProbe, MqttDriverBrowser and
MqttDriver with the constant (string value unchanged).
- Tightens MqttDriverProbeTests.ProbeAsync_EnumAsName: it asserted only
Ok == false + non-empty message, which is also exactly what a JSON-parse
failure produces — so it stayed green under the very regression it names.
It now asserts the probe got past the parse and reached the network.
Falsifiability: deleting JsonStringEnumConverter from MqttJson.Options reddens 9
tests across 4 suites, including the tightened probe test (message becomes
"Config JSON is invalid: The JSON value could not be converted to
MqttProtocolVersion") — which the pre-fix assertions would have passed.
Also references the MQTT driver from Core.Abstractions.Tests so
DriverTypeNamesGuardTests' reflective bin scan discovers the new factory and the
constant/factory parity check stays honest.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Thin typed model over a preserved JsonObject key bag, mirroring the sibling
<Driver>TagConfigModel template. Key names and strictness rules mirror
MqttTagDefinitionFactory rather than inventing an editor-side schema; enums
round-trip as NAMES (the factory reads them strictly by name).
No FullName key: under v3 a tag is identified by its RawPath, which the factory
keys the definition's Name off — a composed identity key here would be dead
weight nothing reads.
Mode is a UI-only sub-shape selector inferred from the blob (any Sparkplug
descriptor key present) and never serialised — the driver takes Plain vs
SparkplugB from its DRIVER config. Only Plain Validate() is implemented; the
Sparkplug branch is a stub Task 24 fills, and its keys are preserved untouched.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Wires the bespoke MqttDriverBrowser into AdminUI's IDriverBrowser
registrations alongside OpcUaClient/Galaxy, overriding the universal
DiscoveryDriverBrowser fallback for driverType "Mqtt".
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Add DriverTypeNames.Sql (+ All) as the shared source of truth, and wire the driver
into the Host: register the factory in DriverFactoryBootstrap.Register via
Driver.Sql.SqlDriverFactoryExtensions.Register(registry, loggerFactory), add the
SqlProbe (= Driver.Sql.SqlDriverProbe) probe via TryAddEnumerable, and add the
Driver.Sql project reference to the Host. Default tier A (SQL client is managed +
cross-platform, so ShouldStub needs no change).
Repoint the interim SqlDriver.DriverTypeName / SqlDriverFactoryExtensions.DriverTypeName
literals at DriverTypeNames.Sql; both still resolve to "Sql", so the AdminUI
TagConfigEditorMap/TagConfigValidator keys and the DriverTypeName-parity test stay
green. The Core guard test discovers factories from its own bin, so it also gains a
Driver.Sql project reference — that is the "registered-factory side" that lets
DriverTypeNamesGuardTests' bidirectional-parity check pass (4/4 green).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
SqlTagConfigModel + SqlRowSelectorModel mirror SqlEquipmentTagParser's
accept/reject boundary byte-for-byte on the fields they own: model/type
serialize as NAME strings (never numbers), KeyValue requires
table+keyColumn+keyValue+valueColumn, WideRow requires table+columnName+a
selector (where-pair OR topByTimestamp), Query is rejected, and unknown keys
(top-level and nested rowSelector) survive load->save. Registered in
TagConfigEditorMap + TagConfigValidator keyed off SqlDriver.DriverTypeName
(DriverTypeNames.Sql is deferred to Task 11). A minimal placeholder
SqlTagConfigEditor.razor lands the map entry now; Task 20 fleshes out the UI.
The load-bearing test rounds editor output back through
SqlEquipmentTagParser.TryParse (editor-output <=> parser-input agreement).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Register SqlDriverBrowser as an IDriverBrowser beside the OpcUaClient/Galaxy
lines, project-reference Driver.Sql.Browser, and add SqlAddressPickerBody.razor:
a DriverBrowseTree schema/table/column picker (DriverOperator-gated) with a
column attribute side-panel, manual-entry model/selector fields retained, that
composes the per-tag TagConfig blob (KeyValue / WideRow) matched to
SqlEquipmentTagParser. Pasted ad-hoc connection strings are session-only and
never persisted into the composed blob or driver config.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
ClusterPrimaryHealthCheck was written three days' worth of debugging ago because
the shared ActiveNodeHealthCheck selected by RoleLeader and reported Healthy for
any node lacking the role. Health 0.3.0 fixes the shared check — oldest Up
member, role-preference scoping, Unhealthy when the node owns no active work —
so the private copy is now duplication rather than a workaround, and it is
deleted.
RolePreference [admin, driver] reproduces exactly what it did: a fused central
node answers for admin (where the singletons and the AdminUI are pinned), a
driver-only site node answers for driver.
SelectOldestUpMemberOfRole now delegates to the shared ClusterActiveNode instead
of re-implementing the age ordering. This is the point of the change rather than
a tidy-up: the redundancy snapshot drives the OPC UA ServiceLevel 250/240 split
while the health tier drives Traefik's admin routing, so two copies of "oldest Up
member of a role" would be two chances to advertise one node as authoritative
while gating the data plane on another. ControlPlane takes a ZB.MOM.WW.Health.Akka
reference for it; the layering trade-off is recorded at the PackageReference.
Behaviour note: a node carrying neither admin nor driver now answers 503 on the
active tier instead of 200. That case is reachable — RoleParser admits dev-only
and cluster-role-only nodes — and 503 is correct, since such a node owns no
active work and must not be in an active-tier pool. No docker-dev node is
affected: every rig node is admin+driver or driver-only.
Tests replaced in kind. The rule itself is now pinned in the library against a
real two-node cluster; what stays OtOpcUa's own decision is the wiring, so
ActiveTierRegistrationTests pins which check is on the active tag, that it is
registered on driver-only nodes too, and that it is not on the ready tier.
Verified: Host.Tests 25/25, ControlPlane redundancy 15/15.
Closes the remaining half of #494, and corrects the half fixed in 8dd9da7d.
The shared ActiveNodeHealthCheck(role: "admin") answered the wrong question twice
over. It returns Healthy for any node LACKING the role, so all four driver-only
site nodes called themselves active and no consumer could find the Primary of a
site Cluster. And it selects by RoleLeader - the lowest-ADDRESSED member - which
is not where Akka places singletons.
Replaced with ClusterPrimaryHealthCheck ("cluster-primary"). One rule: this node
is active iff it is the OLDEST Up member carrying its own active role, where the
active role is admin when the node has it and driver otherwise. That serves both
consumers correctly - a fused admin node answers for admin, so Traefik pins the
AdminUI to the node hosting the singletons; a driver-only site node answers for
driver, which is exactly SelectDriverPrimary, the same election behind
IsDriverPrimary and the OPC UA ServiceLevel 250/240 split. Per-mesh scoping is
free after Phase 6: ClusterState.Members already contains only this node's own
Cluster.
SelectDriverPrimary is generalised to SelectOldestUpMemberOfRole so the
age-ordering rule has one implementation. Two copies of "oldest Up member of a
role" would be two chances to silently disagree about who is in charge, and the
tier and the redundancy snapshot must never disagree.
THE ROLE-LEADER HALF MATTERED IN PRACTICE, not just in theory. On the rebuilt rig
the two orderings diverge right now:
akka leader (lowest address) = central-1
oldest admin member = central-2 <- hosts the singletons
8dd9da7d made the tier a real 503 but still selected by RoleLeader, so it would
have pinned Traefik to central-1 - the node NOT hosting the work. With this change
Traefik correctly routes to central-2.
Live-verified, exactly one 200 per mesh:
MAIN central-2 200 central-1 503 traefik: central-2 UP, central-1 DOWN
SITE-A site-a-1 200 site-a-2 503
SITE-B site-b-1 200 site-b-2 503
Tests: role-selection matrix and the startup-safe Degraded path in Host.Tests
(26 pass); parity between the tier's selector and the redundancy snapshot's, plus
role scoping, added to RedundancyPrimaryElectionTests, which forms a real two-node
cluster deliberately built so the oldest member is not the lowest-addressed one
(5 pass).
Closes the residual Phase-7 gap: when BOTH nodes of a 2-node pair cold-start at the
same instant, each self-first seed runs FirstSeedNodeProcess, times out waiting for
the other, and forms its own 1-node cluster — two Primaries in one pair (the Phase 6
live gate reproduced this reliably on docker). The prior mitigation was operational
(staggered start / compose depends_on), which does not exist on production hardware.
The guard (dark switch Cluster:BootstrapGuard:Enabled, default OFF) prevents the split
without giving up cold-start-alone:
- The lower-address node is the preferred founder: self-first, forms immediately.
- The higher node probes its partner's Akka port (TCP connect) up to PartnerProbeSeconds:
reachable => peer-first (join the founder, never race it); unreachable => self-first
(partner is dead, form alone). The order is decided BEFORE the single JoinSeedNodes,
from an explicit reachability signal — never re-formed mid-handshake (the retired
SelfFormAfter failure mode). When on, Akka gets no config seeds (BuildClusterOptions)
and ClusterBootstrapCoordinator drives the join.
Review-driven hardening: case-insensitive tie-break (a hostname-casing mismatch would
reopen the split); fail-fast validation of the timing knobs; the residual "founder dies
in the probe->join window" hang is documented and made operator-visible (warning + a
restart recovers). Real-ActorSystem coordinator tests cover the load-bearing higher-node
cold-start-alone case; 147/147 Cluster tests pass.
Live-gated on docker-dev (site-a = enablement demo, serialization removed, guard on;
site-b keeps depends_on-serialization + guard off as the A/B control): simultaneous
start -> site-a-1 founds, site-a-2 probes-reachable-joins -> 250/240, NO split;
higher-node-alone -> probes dead founder, self-forms -> 250; founder rejoin -> 240.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 6 wired the re-homed redundancy singleton in Program.cs as
`WithOtOpcUaClusterRedundancySingleton(sp.GetRequiredService<IClusterRoleInfo>())`
inside the AddAkka configurator lambda. That lambda runs WHILE the ActorSystem is
being built, but ClusterRoleInfo depends on the ActorSystem (it is a live
Cluster.State view) — so resolving it there recurses into ActorSystem construction
and every node dies at boot with StackOverflowException. It compiles cleanly (a
runtime DI cycle) and RedundancyStateSingletonRehomeTests passed a hand-built
FakeClusterRoleInfo straight into the extension, mocking around the composition-root
line that overflows. The docker-dev live gate caught it on first boot.
Derive the singleton's cluster-role scope from AkkaClusterOptions (pure config, no
ActorSystem) instead: BuildClusterRedundancySingletonOptions / the extension now take
AkkaClusterOptions and compute Role = Roles.FirstOrDefault(IsClusterRole) ?? driver —
identical to ClusterRoleInfo.ClusterRole, which derives from the same config. Program.cs
passes IOptions<AkkaClusterOptions>.Value, which has no ActorSystem dependency.
Live-verified: rebuilt image boots with zero stack-overflow lines, cluster singletons
form per mesh. 5/5 rehome tests pass; Host builds clean.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Redundancy.md: KNOWN-LIMITATION (per-Akka-cluster election) marked RESOLVED;
new 'Per-cluster meshes (Phase 6)' section + pair-local secrets note; manual-
failover mesh-scope caveat removed; auto-down 1-vs-1 note updated (every mesh is
now two nodes, drill deferred to Phase 7). IManualFailoverService + ClusterRedundancy
razor caveats updated to pair-local reality. Configuration.md documents the
cluster-{ClusterId} role, per-pair self-first seeds, SplitTopologyTransportValidator,
and the intentional Dps default. Program + design status tables + CLAUDE.md marked
Phase 6 DONE.
(Task 8 subagent completed the edits but died on an API error before committing;
reviewed and committed by the controller.)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The per-cluster contact map only gains a key when a row parses, so an
all-malformed cluster has no key at all and is caught by the absent-key arm;
the empty-set guard is defensive/unreachable for the DB producer. Comment-only
(Task 4 code-review minor follow-up).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Task 3 review: scoping only the rows/enabled DB queries by ClusterId left
the observed set (from Cluster.State.Members) filtered by the driver role
alone. In the mixed-mesh window — DB scoping live via a cluster role, but
the physical Akka mesh not yet split so central still sees foreign members
via gossip — a foreign-cluster driver member stays in observed while its row
is filtered out, hitting the RunningNodeHasNoRow branch, which logs at ERROR.
That traded the Warning-level false positive for a worse Error-level one.
Filter observed by the node's own cluster ROLE too. Extracted the projection
into a static FilterOwnClusterDriverMembers(members, ownClusterRole) so the
membership filter is unit-testable without seeding a live cluster (the actor
harness joins as admin only, so observed is always empty). Sourced at
registration from IClusterRoleInfo.ClusterRole alongside ClusterId. Null role
(legacy, no cluster role) keeps every driver member — reconcile-all, unchanged.
Also clarified the ownClusterId doc wording (non-null-and-non-empty).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Review follow-ups on the redundancy re-home (Phase 6, Task 2):
- Program.cs: the hasDriver comment claimed IClusterRoleInfo "throws at host
start if this driver node carries no cluster role" — false since the fallback
landed. Rewrite it to describe the cluster-{ClusterId} scope with the driver-role
fallback for legacy single-mesh / not-yet-migrated nodes.
- ClusterRoleInfo.cs: the SubscriberActor comment described RedundancyStateActor as
the "admin-role singleton" — stale. Note it is now a cluster-{ClusterId}-scoped
singleton spawned on every driver node (LeaderChanged stays a no-op here).
- Add a host/registry-level test for the driver-role FALLBACK path (roles ["driver"],
no cluster role) closing the coverage asymmetry — it was proven only in the pure
helper. Asserts the node boots and registers RedundancyStateActorKey, i.e. the boot
the earlier throw-based version would have aborted.
The boot helper now supplies an in-memory IDbContextFactory and a fake IClusterRoleInfo:
Akka.Hosting invokes singleton props factories eagerly at StartAsync, and the admin
ClusterNodeAddressReconciler factory reads both (the latter for its per-cluster reconcile
scope, added by a sibling Phase 6 task) — without them the admin boot NREs.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
After the fleet splits into one Akka mesh per application Cluster, a
ClusterClientReceptionist serves only its own mesh — so the single
fleet-wide client's SendToAll reached only the mesh whose receptionist
answered, leaving every other cluster silently on its old configuration.
Central now holds one ClusterClient per ClusterId and fans SendToAll
across all of them. LoadContactsFromDb selects ClusterId and groups the
receptionist contacts per cluster (keeping the per-row TryParse guard and
the enabled/non-maintenance filter); ContactsLoaded carries
ContactsByCluster; HandleContactsLoaded diffs per cluster (rebuild changed,
stop+drop vanished, warn-don't-create on empty); RebuildClient builds a
per-cluster client under a clusterId-derived actor name. IMeshClusterClientFactory.Create
gained a clusterId parameter so the per-cluster clients get distinct,
diagnosable names. ApplyAck handling and the DPS branch are unchanged; the
deploy path stays payload-free.
Tests: focused unit coverage of the grouping + fan-out (one-client-per-cluster,
fan-out SendToAll to every cluster client) via the recording factory double,
plus the real two-mesh boundary test extended to central + two separate site
meshes proving one dispatch reaches both and a client scoped to one cluster
never crosses into another. Red-before-green verified: crippling the fan-out
to a single client fails the reaches-both-meshes assertion.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 6 splits the fleet into one mesh per Cluster, so an admin (central)
node's Cluster.State.Members shows only its own pair — it no longer sees
site members via gossip. The ClusterNodeAddressReconciler singleton compared
live membership against EVERY ClusterNode row fleet-wide, so post-split every
foreign-cluster row would log EnabledRowNotInCluster forever.
Scope the actor's two DB queries to rows whose ClusterId matches the admin
node's own (IClusterRoleInfo.ClusterId, sourced at registration). A legacy
admin node with no cluster role (null ClusterId) still reconciles the whole
fleet — it genuinely sees every member via gossip. The pure Reconcile
function and AddressMismatchKind semantics are unchanged; scoping lives
entirely in the actor's queries.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 6 splits the single fleet-wide Akka mesh into one 2-node mesh per
application Cluster. RedundancyStateActor was an admin-scoped cluster singleton
that elected ONE fleet-wide driver Primary — but after the split a driver-only
site pair has no admin node in its mesh to host it, so its Primary would never
be elected.
Re-home it: remove the redundancy WithSingleton block from the admin
WithOtOpcUaControlPlaneSingletons set, and add WithOtOpcUaClusterRedundancySingleton,
a singleton spawned from Program.cs's hasDriver branch on EVERY driver node (the
fused central included). It scopes to the node's own cluster-{ClusterId} role when
present, and falls back to the fixed `driver` role otherwise. Each mesh then has
exactly one, electing its own pair-local Primary and publishing redundancy-state
on its own mesh's DistributedPubSub. The election logic (oldest Up driver member)
and the DPS publish are unchanged — they become pair-local after the split.
The driver-role fallback (Decision 2) deliberately does NOT throw when a node has
no cluster role: that is the pre-Phase-6 fleet-wide behavior on a legacy single
mesh, so legacy/harness (TwoNodeClusterHarness: admin,driver) and not-yet-migrated
deployments keep booting unchanged. On a genuinely split 2-node mesh the `driver`
role is already pair-local (the mesh IS the pair), so the fallback is correct in
both worlds. Cluster-scope, when present, additionally survives an accidental
two-mesh merge by keeping one singleton per cluster.
The role scope + driver-role fallback are pinned by unit tests against the
extracted BuildClusterRedundancySingletonOptions helper (the BuildDowningHocon
pattern); admin-removal + driver-registration are pinned behaviorally against a
booted node's ActorRegistry.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 6 (per-cluster mesh split) needs nodes to carry a cluster-scoped
role like cluster-SITE-A; RoleParser previously rejected anything
outside the fixed admin/driver/dev set. Adds IsClusterRole/
ClusterIdFromRole plus well-known-role constants, and points the three
duplicate "driver" string literals (RedundancyStateActor,
ClusterNodeAddressReconcilerActor, ServiceCollectionExtensions) at the
new RoleParser.Driver so the value has one source, without renaming
any existing DriverRole symbol.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Closes two test-completeness gaps flagged in review of the seam-tap commit — the
ScriptedAlarmHostActor and VirtualTagActor hub emits had no test proving the hub
actually receives them.
- ScriptedAlarm_activation_emits_one_Alarm_item_with_matching_payload: spawns the
host with a capturing fake hub, drives a real engine Inactive->Active transition
through the existing ScriptedAlarms harness, and asserts exactly one
TelemetryItem.Alarm carrying the Activated transition.
- VirtualTag_script_log_emits_one_Script_item_with_same_payload: passes a fake hub
via Props + a publisherFactory, drives an evaluator failure, and asserts a
TelemetryItem.Script carrying the SAME ScriptLogEntry instance handed to the
publisher.
To make the VirtualTagActor tap observable, its hub emit moved to the TOP of
PublishLog, before the test-only publisherFactory early-return branch, so the emit
fires on BOTH the production DPS path and the factory path. Production behavior is
unchanged: production passes publisherFactory: null, so it still reaches the DPS
Tell; the emit just moved a couple lines earlier (both are fire-and-forget).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Each of the four telemetry publish seams now also emits its DPS payload into the
node-local ITelemetryLocalHub (Phase 5), leaving the existing DistributedPubSub
publish untouched:
- driver-health AkkaDriverHealthPublisher -> TelemetryItem.Health
- resilience DriverResilienceStatusPublisherService -> TelemetryItem.Resilience
- alerts ScriptedAlarmHostActor + DriverHostActor (native) -> TelemetryItem.Alarm
- script-logs VirtualTagActor + DpsScriptLogPublisher -> TelemetryItem.Script
DI services take the hub as a required ctor param; actors take an optional
nullable hub threaded through their Props and the real spawn sites
(DriverHostActor <- ServiceCollectionExtensions; VirtualTagActor/ScriptedAlarmHostActor
<- DriverHostActor), so existing test Props constructions keep compiling and simply
do not emit. The hub is a no-op until a gRPC client subscribes, so the emit is safe
on every node.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Joseph Doherty