DriverHostActor.StopChild released the tracker (Invalidate) and evicted the hub synchronously, but
Context.Stop is asynchronous: the stopping child's in-flight SubscribeBulk completed through its
capability invoker AFTER the release, and its RecordCallStart/RecordSuccess/RecordCallComplete
re-created the tracker entry, which DriverResilienceStatusPublisherService then re-emitted every tick
forever. Confirmed with a temporary stack capture in the tracker (removed): every post-Release write
came from DriverInstanceActor.HandleSubscribeAsync -> CapabilityInvoker.ExecuteAsync; the test failed
whenever it landed before the second publisher tick.
- DriverResilienceStatusTracker.Retire (Release) drops the id's snapshots and ignores every later
Record* for it; one guarded write path re-checks after the upsert so a write racing Retire cannot
survive it. Reinstate (called by DriverCapabilityInvokerFactory.Create) lifts it, so the same id
re-added in a later deployment records again.
- TelemetryLocalHub.Evict now also holds the id (late Health/Resilience dropped, neither cached nor
fanned out); ITelemetryLocalHub.Readmit (default no-op), called at the top of SpawnChild, lifts it.
- The removal test is now deterministic: an invoker decorator holds the child's subscribe inside the
real invoker's bracket across the removal and signals when its late writes are done (positive
control) — red 5/5 without the fix, green 20/20 with it. New unit tests for the tracker guard
(incl. a Retire race), the factory Release/Create pair, the hub hold/readmit, and a re-add-same-id
host test.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
MAJOR-1: R4's exact element-type match refused ScadaBridge's CLR-list
writes (its element types are only Boolean/Int32/Float/Double/String/
DateTime) to every Int16/UInt16/Byte/UInt32/Int64/UInt64 array node and a
Double list to a Float array. EvaluateEquipmentWriteStructure now converts
a numeric array of another numeric element type element-wise, CHECKED, into
the node's element type and hands the converted array back through
OnEquipmentTagWrite's ref value, so the node stores and the driver receives
the node's own type. Any out-of-range, fractional-to-integer, NaN/Inf-to-
integer or Float-overflowing element refuses the whole write BadTypeMismatch
(nothing dispatched) - so the UInt32 -> Int32 bit reinterpretation stays
closed. Non-numeric mismatches stay refused. The wrong 'no client relies on
a looser rule' comment is corrected.
MINOR-1: the array-value-to-scalar-node refusal broke writes back to
browse-committed array tags (committed scalar, isArray never set) on
pass-through drivers. It now applies only when the owning driver's scalar
encoder cannot take an array (DriverArrayWriteSupport.
RefusesArrayValueOnScalarTag: Modbus, S7), carried RawTagPlan.
ScalarRefusesArrayValue / UnsReferenceVariable.BackingScalarRefusesArrayValue
-> EnsureVariable(refuseArrayOnScalar) -> a per-node set in the node
manager. OpcUaClient/Galaxy/others keep pre-#610 dispatch. Not on the
classifier whitelist (driver-type edit rebuilds); a mirror whose flag
changes is recreated, not updated in place.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
MAJOR-1: the SDK skips TypeInfo.IsInstanceOfDataType on any node with an
OnWriteValue handler (every writable equipment node), and the node manager's
fail-fast treated numeric<->numeric as compatible, so a UInt32[] to a Galaxy
Int32 array matched MxValueEncoder's int[] arm by CLR covariance and was
written bit-reinterpreted with a Good result. EvaluateEquipmentWriteStructure
now matches an ARRAY write's element type exactly (Byte[]/ByteString the only
equivalence; Variant[]/object[] refused) -> BadTypeMismatch, nothing
dispatched. MxValueEncoder's int[]/long[] arms are exact-type guarded too, and
the Galaxy parity test's false "the SDK refuses" premise is corrected (the
UInt32/UInt64 exclusion is now the encoder's own answer). Scalar writes keep
numeric widening by decision: consumers write CLR-typed values (Double to a
Float node) and every driver's scalar encoder range-checks.
MINOR: Modbus row honours maxRegistersPerWrite (smallest of DriverConfig /
the tag's DeviceConfig, default 123) and FC15's 1968-coil ceiling, and
requires region + numeric address; Modbus/S7/Galaxy require isArray + integer
arrayLength >= 1 (Galaxy: an unbounded node would let a short write resize the
attribute); an array value to a scalar node is refused inline. Docs: Uns.md
array-write rules + upgrade note (ReadWrite array tags become client-writable
on first boot), Modbus.md, Galaxy.md.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
Review R1:
- M1 (#609): DriverHostActor.StopChild releases the instance's resilience state before evicting the hub.
- m2 (#597): DriverHostActor publishes ShouldServiceAsPrimary() to IDriverPrimaryRoleSource at start and on
cluster membership changes (deduped), so the native-ack gate denies in a pair's boot window.
- m4 (#601): ApplyCachedArtifact returns false on empty/malformed bytes; the cached revision is not adopted,
Direct mode goes Stale (no RUNNING FROM CACHE), FetchAndCache lands Steady-with-no-revision and a
re-dispatch of the same revision really applies.
- m3 (#603): LocalDbReplicationPolicy splits into CapturesChanges (PeerAddress||ApiKey, registration) and
SharesRowsWithPeer (ApiKey && (PeerAddress||SyncListenPort>0), alarm drain gate).
- m5 (#618): Program.cs catch-all filters HostAbortedException (design-time tooling).
- m6 (Cluster-002): ClusterRoleInfo seeds only Up/WeaklyUp members.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
Review R1 M2: since #597 the SDK timer fires at real expiry, and OnTimedUnshelve routed an operator-style
Unshelve by 'system' from both pair nodes, pre-empting the engine's AutoUnshelve. The engine owns expiry,
so the callback now routes nothing. Integration test: an engine stand-in that acts on a routed Unshelve and
expires late, plus a wrapped OnTimedUnshelve (positive control: the SDK timer fired).
Review R1 m1: Part9StateMachine.ApplyAcknowledge records a commented re-ack as AddComment (persisted and
replicated) instead of a NoOp; an uncommented re-ack is still a no-op.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
Array nodes were forced read-only for every driver. A new single authority,
Core.Abstractions DriverArrayWriteSupport, says which (driver, element type)
pairs encode a whole-array write: Modbus (Coils / numeric HoldingRegisters),
S7 (Byte/Int16/UInt16/Int32/UInt32/Float32), OpcUaClient (pass-through) and
Galaxy (Boolean/Int32/Int64/Float/Double/String/DateTime). Everything else
stays read-only, so the server never advertises a write the driver refuses.
- Composer derives RawTagPlan.ArrayWritable + UnsReferenceVariable
.BackingArrayWritable at compose time (both seams, byte-parity); the
applier's one writability rule is Writable && (!IsArray || ArrayWritable)
for materialise and surgical paths, raw and UNS alike.
- ArrayWritable is on the classifier's surgical raw-tag whitelist and
mirrored on the UNS record (UnsReferenceBackingParityTests updated).
- Node manager refuses a wrong-length / non-array value on an array node
(BadTypeMismatch) and any IndexRange write (BadWriteNotSupported) inline,
before dispatch; WriteOperate/ACL gates and the revert are unchanged.
- Per-driver parity tests pin the table against the real Modbus write path,
S7 EncodeArrayBlock and Galaxy MxValueEncoder (UInt32/UInt64 excluded:
array covariance reinterprets their bits).
- Over-the-wire proof in OpcUaServer.IntegrationTests (ArrayWriteWireTests).
- ValueRank > 1 documented as out of scope by decision (docs/Uns.md).
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
ClusterRoleInfo resolved the leader address through Cluster.State.Members, which is
published asynchronously; a miss stored null permanently, so RoleLeader(role) stayed null
(flaky test timed out under load: 6/10 failing with CPU contention, 0/20 after). Only the
address is ever read, so store it directly. Adds a deterministic unit test (fails without
the fix) and InternalsVisibleTo for the Cluster.Tests project.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
options.Roles is unconditionally overwritten by AddOtOpcUaCluster's PostConfigure to equal
EffectiveRoles.Gossiped (T2's own structural-identity fix), so comparing options.Roles against
Gossiped could never observe a divergence — an overlay setting Cluster:Roles was silently
ignored, not fail-start. The validator now injects IConfiguration and re-reads the raw
Cluster:Roles section LIVE (reflecting any overlay applied after EffectiveRoles was first
resolved), normalized through EffectiveRoles.Resolve for reuse of its own validation.
AddOtOpcUaCluster now TryAddSingleton(configuration) so a bare ServiceCollection test harness
(not just WebApplicationBuilder) can supply IConfiguration to the validator.
Both guard tests rewritten to go through the real AddOtOpcUaCluster + PostConfigure pipeline
over a live ConfigurationManager, simulating an overlay as a second provider added AFTER
EffectiveRoles was resolved off the same object — exactly Program.cs's own sequencing. Confirmed
red beforehand: the "changes" test failed to throw against the pre-fix validator.
With Security:Acl:Enabled=true, a session holding any grant narrower than
Cluster could not browse the served address space at all — confirmed by the
live gate.
Mechanism. Both realm trees hang under ONE anchor folder minted in
CreateAddressSpace: NodeId("OtOpcUa", <raw ns>). ResolveParentFolder parents
every top-level raw folder AND every top-level UNS area under it, and the
Browse override throws BadUserAccessDenied for the WHOLE call when its source
node is denied. DataPlaneAuthorizer.BuildScope split that identifier like a
path, producing {Kind: Raw, Segments: ["OtOpcUa"]} — but trie grant paths are
["raw", <RawPath segments…>] and "OtOpcUa" is never a path segment, so
PermissionTrie.HasBrowseTransit failed at the realm-child lookup and
CollectMatches yielded only Cluster-scoped grants. Deny, at the front door.
The trie already handled a zero-segment scope as "the realm root, itself an
ancestor of everything granted in that realm" (PermissionTrie.cs:101-118) —
no production path had ever produced that scope. This wires it up.
- V3NodeIds.SharedRootFolderId names the anchor identity; CreateAddressSpace
now mints the NodeId/BrowseName/DisplayName from it (behaviour-identical,
makes the coupling explicit).
- BuildScope maps that identifier (realm-qualified to Raw — in the UNS
namespace s=OtOpcUa is an ordinary Area) onto the zero-segment Raw scope.
- Evaluate adds a second-realm OR-leg for Browse on the anchor only: it fronts
both trees, so a principal whose only Browse grant lives in the UNS tree
still enters through this raw node. Browse only; transit confers navigation,
never read/write/subscribe.
- No trie, evaluator or builder change. No reordering above the decision cache
— key revocation is still re-checked before the lookup — and the cache key
is unchanged (the combined verdict caches under the anchor's own
identifier/realm; a generation advance clears it).
Falsifications actually run (implement-then-revert-the-line, plus one
genuine red-first):
- DataPlaneAuthorizerTests: the 7 new facts were written BEFORE the fix and 4
went red (the 3 deny-only ones passed vacuously, as a deny-only test must).
- Deleting the UNS OR-leg reddens exactly one fact,
The_shared_root_folder_is_browsable_on_a_uns_only_browse_grant (26/27 pass).
- Dropping the realm qualification from IsSharedRootFolder reddens exactly
A_uns_area_named_like_the_anchor_gets_no_anchor_semantics. Its first draft
did NOT catch this — a raw-only grant is refused either way; the
discriminating grant had to be an unrelated UNS one, whose own transit marks
the zero-segment UNS scope the un-qualified mapping would land on.
- Disabling the BuildScope anchor branch reddens the new end-to-end fact
NodeManagerStage6EnforcementTests.The_shared_root_folder_is_browsable_on_a_narrow_grant_and_still_filters
(real booted server, real Browse dispatch: BadUserAccessDenied on the whole
call), which also pins that the reference filter still withholds the
non-granted sibling.
- Core PermissionTrieBrowseTransitTests gains 3 characterization facts pinning
the previously-dead realm-root branch (transit true for the granting realm's
root, false for the other realm's, false without the Browse bit). These were
green on arrival — the trie was always correct; nothing called it.
Tests: Core 351 -> 354 passed; OpcUaServer 482 -> 490 passed. Build clean.
TagConfigJson.ClassifyEnumText classified via Enum.TryParse(text, ignoreCase: true),
which accepts a numeric STRING for any enum: "1" returns the member at that ordinal
and an out-of-range "99" parses just as happily. Every driver contract's strict enum
read inherits that, so a Sql tag authored "type":"99" was ACCEPTED, carried
(DriverDataType)99, and SqlPollReader.TryCoerce fell through its switch to the String
arm — a stringified cell published Good under a type nothing can map.
Refuse a digit- or sign-leading spelling before the parse (judged on the TRIMMED text,
because Enum.TryParse trims), and require Enum.IsDefined — which covers what the
leading-character check cannot, since TryParse also ORs a comma-separated list for a
non-[Flags] enum. Mirrors RawBrowseCommitMapper.ParseNativeType and
MTConnectTagDefinitionFactory.TryParseDataType, which already guard their own parses.
Three AdminUI restatements of the same rule now ask ClassifyEnumText instead of
Enum.TryParse (TagConfigJson.GetEnum, SqlTagConfigModel.ReadEnumStrict,
MqttTagConfigModel.DescribeInvalidEnum/GetEnumNullable) — leaving them would have opened
the #549 direction the moment the runtime tightened: a blob the modal accepts and the
driver skips.
RED first: 10 failing in Core.Abstractions.Tests, 4 in Driver.Sql.Tests, and 10 of the
11 new fleet-wide parity cases (re-confirmed by reverting the Core fix).
The connect path was the one capability call the resilience pipeline never
saw, so a driver that could not connect contributed zero breaker input and
the /hosts resilience panel read 'All targets healthy' against a dead
endpoint. InitializeAsync now routes through the invoker as
DriverCapability.Initialize, keyed on the driver-instance id.
- No pipeline retry: the actor's RetryConnect timer is the retry loop
(parser forces retryCount overrides to 0, same funnel as write-shaped).
- Breaker threshold 2, not the Read-sized 5: the breaker samples 30 s and
the reconnect timer fires every 10 s, so at most three attempts fit one
window — threshold 5 could never open. Pinned by its own test.
- An open breaker fast-fails into the same InitializeFailed path as a real
connect failure; the 15 s break window costs at most one extra tick.
The best-effort teardown ShutdownAsync stays unwrapped.
- Companion fix: DriverCapabilityInvokerFactory.Create now invalidates the
instance's tracker snapshots with its pipelines — a respawn during an
outage discarded the only breaker whose OnClosed could clear the OPEN
flag, stranding a false 'Breaker OPEN' chip forever.
Live gate PASSED on docker-dev (image symbol-verified per container):
rtu-gate against dead 10.100.0.35:5021 opened its Initialize breaker on
both centrals with the red 'BREAKER OPEN' chip on /hosts; a socat
5021→5020 forwarder recovered it (both breakers closed within 1 s, panel
back to 'All targets healthy'); healthy MAIN-modbus recorded zero breaker
events. Full suite 50/50 green. Evidence:
docs/plans/2026-08-07-adminui-followups-and-live-gates.md §7.
MeshGrpcBoundaryTests replaces the MeshClusterClientBoundaryTests deleted in step 1. Three
self-seeded single-member clusters sharing the ActorSystem name, joined ONLY by gRPC, driving
the REAL CentralCommunicationActor -> MeshGrpcCommandDialer -> Kestrel h2c ->
MeshCommandAuthInterceptor -> NodeCommandGrpcService -> NodeCommunicationActor -> EventStream:
1. No_cross_mesh_Akka_association_forms_between_central_and_either_site — the program's own
point, kept as a test rather than a live-gate note: each cluster stays single-member and
none ever holds another's address as a member.
2. One_dispatch_through_central_reaches_a_node_in_every_cluster_mesh — the fan, with each
site holding its OWN per-cluster PSK, so a dial that selected the wrong cluster's key
would be refused at the far side.
3. Falsifiability_control_a_dial_carrying_the_wrong_cluster_key_delivers_nothing — without
it, (2) cannot distinguish "the gRPC boundary carried it" from "something else in the
process did". The same node then accepts the right key, so the silence is the auth gate
and not a dead endpoint.
All three run fully offline — no Docker fixture — and pass.
Comment sweep across every remaining clusterclient|receptionist hit in src/ and tests/:
the program label becomes "mesh gRPC command plane", and every comment that described the
deleted transport as live now describes what actually runs (Dps = pair-local default,
Grpc = cross-mesh). mesh_transport.proto's header/service comments are reworded; no message,
field or enum is touched — the contract is locked. MeshCommActorPathTests is reshaped rather
than retired: the two paths still exist and still matter, but what a mismatch costs is now a
misleading Unavailable from the gRPC ingress rather than a silently dropped receptionist send.
The only surviving mentions of the old names are the ones that must survive: the fail-closed
rules that name "ClusterClient" as a retired value refused at boot, their tests, and this
test class's own account of what it replaces.
MeshTransportOptions.ModeClusterClient is gone. MeshTransportOptionsValidator accepts Dps or
Grpc and nothing else, so a node left carrying the string "ClusterClient" is now REFUSED at
boot naming MeshTransport:Mode and both remaining values — deliberately fail-closed rather
than falling back to Dps, which on a split-topology node would deliver nothing at all. A new
fact pins exactly that.
SplitTopologyTransportValidator tightens from an allow-list of two to Grpc only, as its own
comment said it would once ClusterClient was deleted. Dps on a cluster-role node stays
refused; the rule is still an explicit allow-list, so a typo'd mode fails too.
Tests: the two ClusterClient-mode facts in MeshTransportOptionsValidatorTests are deleted
(their Dps siblings already assert the same rule for the one remaining non-Grpc mode), the
SplitTopology fixtures move to Grpc, and the three-mode Theories in
ControlPlaneStarterOrderTests / CentralCommunicationActorTests become two-mode.
The property, the four validator rules that graded it (empty / non-akka.tcp / actor-path
suffix / one-bad-among-good) and all twelve docker-compose entries (6 nodes x 2) are gone.
The compose comment blocks that explained "/system/receptionist is appended at construction
time" and the client's contact rotation go with them, and the surrounding prose now
describes the Grpc command plane it actually runs.
CentralGrpcEndpoints' doc no longer says it "replaces" the Akka contact points — there is
nothing left to replace — and ContactRefreshSeconds' failure message names the node
dispatch set rather than the ClusterClient contact points.
`rg CentralContactPoints src tests docker-dev` is now empty.
akka.extensions loses ClusterClientReceptionistExtensionProvider and the whole
akka.cluster.client block (buffer-size / reconnect-timeout / receptionist.role) goes with
it. The DistributedPubSub extension and the maximum-frame-size / log-frame-size-exceeding
canary both stay — DPS is the pair-local transport and the frame canary guards every
remoting message, neither of which the cross-mesh deletion touches.
MeshTransportHoconTests drops its three ClusterClientSettings/ClusterReceptionistSettings
asserts and keeps the frame-size pin and the DPS-extension pin, renamed since it no longer
has a receptionist to survive alongside.
Two review items from Task 1B.1, both diagnostic:
1. CentralCommandGrpcService's "actor not ready" Warning omitted the peer and
the correlation id while the decode-failure Warning six lines above carried
both. These are the only two ways an ack fails to reach the coordinator, and
an operator has to correlate either against the node's own log. Decoding has
already succeeded on this path, so the DECODED correlation id is free.
2. A fused driver,admin,cluster-MAIN node under Mode=Grpc owes BOTH validator
surfaces at once (driver keys AND ConfigServe:GrpcListenPort). The two blocks
are independent `if`s; chained as `else if` a fused central node would be
graded on one surface only and boot with nowhere for any ack to land. Pinned
from both sides plus the fully-configured pass.
Joseph Doherty