9227d03ca88a551217fe022b753ae923ef565672
275 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Joseph Doherty
|
a107a1a8b9 |
feat(mtconnect): host registration + DriverTypeNames const + guard parity (Task 16)
Register the MTConnect factory + probe with the Host, add the DriverTypeNames.MTConnect constant, and keep DriverTypeNamesGuardTests green. - DriverTypeNames: new MTConnect const + appended to All. - DriverFactoryBootstrap: MTConnectDriverFactoryExtensions.Register in Register(...) at the default Tier A (managed HttpClient/XML, in-process; Tier C is the only tier arming process-recycle, wrong here), and TryAddEnumerable(IDriverProbe -> MTConnectDriverProbe) in AddOtOpcUaDriverProbes so the probe reaches admin-only nodes (the admin-pinned Test-Connect singleton) without double-registering on a fused admin,driver node. - Host.csproj: ProjectReference to Driver.MTConnect (required to compile the Register call). - Core.Abstractions.Tests.csproj: ProjectReference to Driver.MTConnect so the guard's bin scan discovers the factory. This and the constant MUST land together -- verified RED (2/4 facts) with the constant alone. - DriverProbeRegistrationTests: MTConnect added to AdminUiDriverTypeKeys; its idempotency fact asserts distinct-probe-count and goes RED without this (and RED if TryAddEnumerable is downgraded to AddSingleton). Reconciles all four "MTConnect" literals onto the one constant: the factory's DriverTypeName, MTConnectDriver.DriverType, and MTConnectDriverProbe.DriverType now all alias DriverTypeNames.MTConnect (no circular reference -- Core.Abstractions is a leaf both driver projects already reference). This is the repo's TwinCat/Focas anti-drift convention. |
||
|
Joseph Doherty
|
c3a2b0f773 |
Merge branch 'feat/mqtt-sparkplug-driver'
# Conflicts: # src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/DriverTypeNames.cs # src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Shared/Drivers/DriverConfigModal.razor # src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/EndpointRouteBuilderExtensions.cs # src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Uns/TagEditors/TagConfigEditorMap.cs # src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Uns/TagEditors/TagConfigValidator.cs # src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/ZB.MOM.WW.OtOpcUa.AdminUI.csproj # src/Server/ZB.MOM.WW.OtOpcUa.Host/Drivers/DriverFactoryBootstrap.cs |
||
|
Joseph Doherty
|
2193d9f2e4 |
fix(config): refuse to store a Sql credential in a node config override (#499)
`ClusterNode.DriverConfigOverridesJson` is the third surface a Sql driver's config is persisted on, and the only one nothing gated. It is a map keyed by `DriverInstanceId` merged onto the cluster-level `DriverConfig`, so a literal `connectionString` pasted there leaks exactly as one pasted into the driver — the leak #498 closed for `DriverInstance.DriverConfig` and `Device.DeviceConfig`. Gated at the SAVE, not at the deploy — a deliberate departure from both options the issue offered: - `DraftSnapshot` carries no `ClusterNode` rows, and adding them would widen the snapshot and every builder of one for a single rule about a value that never enters the artifact. - More to the point, a deploy gate is the wrong instrument here. Node overrides are not in the artifact, so blocking a deploy would not stop the credential being stored — it is already in the database and replicated by then. Refusing the save is the only point where "refuse to store it" is literally true, which is #498's own framing: discarding a secret on read is not the same as refusing to store it. The check moves into a shared `SqlCredentialGuard` so the two enforcement points cannot drift; `DraftValidator` now calls it instead of its own private helper. Kept deliberately narrow — the `Sql` driver's `connectionString` only, and only for instance ids that ARE Sql drivers. The issue asked whether to generalise to credential-shaped keys across every driver type; not doing that, for the same reason #498 did not: a broader sweep would start refusing configs that are legitimate today for drivers which never made an indirect-credential guarantee, which is a regression rather than defence in depth. Widen per driver, as each gains its own contract. The error names the offending driver instance(s) and NEVER the value — it reaches the AdminUI and the audit trail. 14 new cases cover both halves, including the case variants (System.Text.Json binds `ConnectionString` to `connectionString`, so a case variant is the same key, not a bypass), non-Sql ids being ignored, and every blank/malformed/non-object shape. |
||
|
Joseph Doherty
|
549c656489 |
fix(alarms): re-assert non-normal scripted-alarm conditions after a (re)load (#487)
A deploy that triggers a FULL address-space rebuild clears `_alarmConditions` and re-materialises every condition node fresh — inactive, acked, confirmed, unshelved. The engine reloads its persisted state and re-derives Active from the predicate, so it ends up correct; but there is no transition to report, so `LoadAsync` yields `EmissionKind.None` and `OnEngineEmission` drops it. Nothing writes the node. Live-reproduced on the docker-dev rig against the pre-fix image: an alarm that had fired and cleared but was still UNACKNOWLEDGED came back from a rebuild reporting Acknowledged with Retain=false — so it disappeared from ConditionRefresh entirely, while the engine still held Unacknowledged. An operator's outstanding alarm silently vanishes from the alarm list on deploy. `ScriptedAlarmHostActor.ReassertConditionNodes` closes it. After each load it reads the new `ScriptedAlarmEngine.GetProjections()` and sends one `AlarmStateUpdate` per alarm NOT in the Part 9 no-event position. Load-bearing properties: - Node-only. It sends `AlarmStateUpdate` and nothing else — never the `alerts` topic, never the telemetry hub. `alerts` is the historization path, so a row per alarm per deploy would append a duplicate historian/AVEVA record every time anyone deploys. This is why it reads a projection rather than re-emitting: `ScriptedAlarmProjection` carries no `EmissionKind`, so there is nothing for the emission machinery — or a future refactor — to mistake for a transition. The issue's sketch said `GetState(alarmId)` + `LoadedAlarmIds`; that alone would have clobbered the node's severity and message, so the projection also carries the definition's severity, the resolved message template, and the last-observed worst-input quality. - Only non-normal alarms. One in the no-event position already matches what the materialise built, so a never-fired alarm stays byte-identical to a deploy without this pass (including its Message, which the materialise seeds with the display name), and an all-normal deploy writes nothing at all. - Timestamp is the condition's own `LastTransitionUtc`, not the deploy instant. - No duplicate Part 9 events: `WriteAlarmCondition`'s delta-gate compares against the node's CURRENT state, so a re-assert onto a freshly materialised node is a genuine delta (fires once — correct, the rebuild reset what clients see) while one onto a surgically-preserved node is suppressed. Tests: 6 new host tests + 6 new engine tests. Falsifiability checked by disabling the call — 4 of the 6 host tests go red; the other 2 are absence guards against an over-broad fix and pass either way by design. Live gate (docker-dev, central-2, pre-fix image vs. rebuilt image): - pre-fix: condition absent from ConditionRefresh after a rebuild; - post-fix: present, Unacknowledged, Retain=True, stamped with its own LastTransitionUtc rather than the restart instant; - /alerts stayed empty across two re-asserts, with the panel proven live by a real ACTIVATED transition immediately afterwards — no duplicate history. |
||
|
Joseph Doherty
|
1919a8e538 |
feat(config): reject a persisted Sql connectionString at the deploy gate (#498)
The Sql driver's "a pasted literal connection string is never persisted" guarantee was
enforced only on the READ path: SqlDriverConfigDto has no connectionString property and
UnmappedMemberHandling.Skip drops the key on deserialization. Nothing stopped it being
WRITTEN. Config blobs are schemaless JSON columns and the fallback authoring pattern for
a driver without a typed form is a raw-JSON textarea, so an operator pasting
{"connectionString":"Server=...;Password=..."} would put a live credential in the
ConfigDb — persisted, replicated to every node's artifact cache, readable by anyone with
config access — while the runtime silently ignored it and the driver failed to connect.
Discarding a secret on read is not the same as refusing to store it.
DraftValidator.ValidateSqlConnectionStringNotPersisted fails the deploy
(SqlConnectionStringPersisted) when a Sql-typed driver instance's DriverConfig, or the
DeviceConfig of any device beneath it, carries a top-level connectionString key.
- Both surfaces are checked because DeviceConfig is merged onto DriverConfig before the
DTO sees it — a credential pasted into the device blob is the identical leak.
- The key is matched case-insensitively: System.Text.Json binds ConnectionString to a
connectionString property by default, so an ordinal match would leave the obvious
bypass open.
- Only the top level is scanned; the DTO is flat, so a nested occurrence cannot bind and
is not the mistake this rule exists to catch.
- The message never echoes the value — validation errors reach the AdminUI, the deploy
log and the audit trail, and repeating the string there would leak the credential the
rule is refusing to store. Pinned by a test.
- Malformed/blank/non-object JSON never throws: a parse failure would take down every
other rule in the same pass.
16 tests. Verified falsifiable — with the rule disabled the 6 positive assertions fail
and the 10 negatives stay green, so none of them passes vacuously.
Design §8.2's "if an inline connectionString is ever permitted (dev convenience only),
the AdminUI redacts it" carve-out is withdrawn in the same change: redaction only hides a
credential that has already been written.
Not covered, and filed as a follow-up: ClusterNode.DriverConfigOverridesJson is a third
persistence surface for a driver config blob, but it is not part of DraftSnapshot, so
this validator cannot see it.
|
||
|
Joseph Doherty
|
8d9155682d |
fix(mqtt): browse-commit writes a bindable address + a Request-rebirth affordance
Two gaps the Task 23/24 review found, both blocking Task 26's live gate.
Gap 1 — browse-commit produced a silently dead MQTT tag. RawBrowseCommitMapper
had no `Mqtt` case, so a committed leaf fell through to the generic
`{"address": …}` key. Neither MqttTagDefinitionFactory entry point reads
`address`: the tag deployed clean and reported BadNodeIdUnknown forever, with
no signal at commit time. Predates Task 23 (Plain was affected too), but Task 23
built the Sparkplug metric tree precisely so an operator could browse and commit
a binding.
The address is a DESCRIPTOR, not a reference string, and it cannot be recovered
from the browse node id: `{group}/{node}[/{device}]::{metric}` where a metric
name legitimately contains `/` (`Node Control/Rebirth`) — the ambiguity
MetricSeparator's remarks already name. So the session STATES it, via a new
`AttributeInfo.AddressFields` seam, and the mapper reads it. Which keys are
emitted is also how the mapper learns Plain vs Sparkplug — the driver type
reaching it is just `Mqtt`, and the mode lives on the driver config. Key names
are single-sourced in the new `MqttTagConfigKeys` (producer, mapper, factory),
with the literals pinned by a test so a symmetric rename cannot silently unbind
already-persisted blobs. A leaf with no stated address is refused at commit in
words rather than committed dead.
Gap 2 — RequestRebirthAsync had no UI. Task 23 shipped it backend-only; Task 26
step 1 assumes the button, and Task 25's runbook notes the picker tree stays
empty until a birth lands, so it is the only way to fill it on demand. Added to
the /raw browse modal: scope = the clicked tree node (device/metric resolve up to
their edge node, group fans out and is refused whole past 32), an explicit
two-click confirm naming the resolved scope and its consequence, the outcome or
the refusal shown rather than swallowed. Offered only for a Sparkplug session
(new `IRebirthCapableBrowseSession.RebirthAvailable` — one session class serves
both modes, so a type test alone would draw a button that can only throw) and
only to a DriverOperator; the server-side gate remains the boundary.
Tests: MQTT 545 → 581, AdminUI 781 → 800. The round-trip tests feed the emitted
blob to the REAL factory and were falsified by mutating the emitted key name.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
|
||
|
Joseph Doherty
|
64ec7aa43a |
feat(mqtt): Sparkplug birth-driven browser tree + scoped Request-rebirth action
Unseals the Sparkplug branch of the MQTT address-picker browser that Task 10 deliberately sealed shut, and adds the first (and only) sanctioned publish on a browse session. - MqttBrowseSession serves Group -> EdgeNode -> [Device] -> Metric in Sparkplug mode, decoded from observed NBIRTH/DBIRTH certificates (the only Sparkplug messages that name their metrics). Node-level metrics hang directly under their edge node -- no synthesised device folder, because the authored address tuple genuinely has deviceId = null for them. Metric node ids use a '::' separator: a metric name may contain '/', so slash-joining would make a node-level metric indistinguishable from a device folder. - AttributesAsync is purely birth-derived in Sparkplug mode. The plain path's UTF-8 inference / payload-snippet machinery does not run and no payload bytes are retained: a Sparkplug body is protobuf and would be reported as "(N bytes, binary)" for every metric in the plant. A datatype ToDriverDataType cannot map is reported as the Sparkplug type name, never coerced. - RequestRebirthAsync(scope) is the one publishing member, reached only by an explicit operator action. It routes through the counted PublishAsync chokepoint via a private RebirthTransport adapter, so PublishCountForTest still proves that OpenAsync/RootAsync/ExpandAsync/AttributesAsync/Observe/ DisposeAsync publish nothing -- now in both modes. A device or metric scope resolves up to its owning edge node (NCMD is node-addressed); group scope enumerates observed edge nodes and is refused whole past MaxGroupRebirthNodes = 32. - BrowserSessionService.RequestRebirthAsync gates on the same DriverOperator policy that gates the picker, evaluated server-side where the action happens rather than only at render time, fails closed, and Info-logs the scope and the published count. Exposed through the new IRebirthCapableBrowseSession contract so "does this session write?" stays a type question. - ToBrowseOptions' Last-Will landmine re-verified: MqttDriverOptions still carries nothing will-shaped (an NDEATH is a broker-published will and would be invisible to PublishCountForTest), now pinned by a reflection guard. Falsifiability: injecting a publish into RootAsync reddens the Sparkplug and plain passivity tests; double-publishing per target reddens all four one-NCMD-per-node assertions. 572/572 MQTT tests, 24/24 AdminUI browsing tests. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
f79d13e2d8 |
feat(mqtt): factory + DriverTypeNames.Mqtt + host factory/probe registration
Task 9. Adds MqttDriverFactoryExtensions (DriverTypeName = DriverTypeNames.Mqtt, direct deserialization of MqttDriverOptions — no intermediate DTO, mirroring OpcUaClientDriverFactoryExtensions), the DriverTypeNames.Mqtt constant, and both host wiring sites: the factory in DriverFactoryBootstrap.Register and the probe in AddOtOpcUaDriverProbes (the admin-node path Program.cs calls in its hasAdmin block — a probe wired only on driver nodes makes Test-connect silently dead). Carried-forward items: - Converges every MQTT config-parsing seam onto ONE JsonSerializerOptions — MqttJson.Options, in .Contracts alongside MqttDriverOptions. It replaces the probe's internal JsonOpts and the browser's separate private copy; the factory, the probe, MqttDriver.ParseOptions and MqttDriverBrowser now all parse through it. .Contracts is the only assembly all four consumers reference, and the browser's reference to the runtime .Driver project is a documented layering exception scheduled for removal — anchoring the options there would resurrect the duplicate the day it goes away. - Replaces the "Mqtt" literals in MqttDriverProbe, MqttDriverBrowser and MqttDriver with the constant (string value unchanged). - Tightens MqttDriverProbeTests.ProbeAsync_EnumAsName: it asserted only Ok == false + non-empty message, which is also exactly what a JSON-parse failure produces — so it stayed green under the very regression it names. It now asserts the probe got past the parse and reached the network. Falsifiability: deleting JsonStringEnumConverter from MqttJson.Options reddens 9 tests across 4 suites, including the tightened probe test (message becomes "Config JSON is invalid: The JSON value could not be converted to MqttProtocolVersion") — which the pre-fix assertions would have passed. Also references the MQTT driver from Core.Abstractions.Tests so DriverTypeNamesGuardTests' reflective bin scan discovers the new factory and the constant/factory parity check stays honest. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
9b5a96876e |
feat(sql): register Sql factory + probe in Host, add DriverTypeNames.Sql
Add DriverTypeNames.Sql (+ All) as the shared source of truth, and wire the driver into the Host: register the factory in DriverFactoryBootstrap.Register via Driver.Sql.SqlDriverFactoryExtensions.Register(registry, loggerFactory), add the SqlProbe (= Driver.Sql.SqlDriverProbe) probe via TryAddEnumerable, and add the Driver.Sql project reference to the Host. Default tier A (SQL client is managed + cross-platform, so ShouldStub needs no change). Repoint the interim SqlDriver.DriverTypeName / SqlDriverFactoryExtensions.DriverTypeName literals at DriverTypeNames.Sql; both still resolve to "Sql", so the AdminUI TagConfigEditorMap/TagConfigValidator keys and the DriverTypeName-parity test stay green. The Core guard test discovers factories from its own bin, so it also gains a Driver.Sql project reference — that is the "registered-factory side" that lets DriverTypeNamesGuardTests' bidirectional-parity check pass (4/4 green). Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
279d1d0fb1 |
feat(cluster): simultaneous-cold-start split-brain guard (opt-in)
Closes the residual Phase-7 gap: when BOTH nodes of a 2-node pair cold-start at the same instant, each self-first seed runs FirstSeedNodeProcess, times out waiting for the other, and forms its own 1-node cluster — two Primaries in one pair (the Phase 6 live gate reproduced this reliably on docker). The prior mitigation was operational (staggered start / compose depends_on), which does not exist on production hardware. The guard (dark switch Cluster:BootstrapGuard:Enabled, default OFF) prevents the split without giving up cold-start-alone: - The lower-address node is the preferred founder: self-first, forms immediately. - The higher node probes its partner's Akka port (TCP connect) up to PartnerProbeSeconds: reachable => peer-first (join the founder, never race it); unreachable => self-first (partner is dead, form alone). The order is decided BEFORE the single JoinSeedNodes, from an explicit reachability signal — never re-formed mid-handshake (the retired SelfFormAfter failure mode). When on, Akka gets no config seeds (BuildClusterOptions) and ClusterBootstrapCoordinator drives the join. Review-driven hardening: case-insensitive tie-break (a hostname-casing mismatch would reopen the split); fail-fast validation of the timing knobs; the residual "founder dies in the probe->join window" hang is documented and made operator-visible (warning + a restart recovers). Real-ActorSystem coordinator tests cover the load-bearing higher-node cold-start-alone case; 147/147 Cluster tests pass. Live-gated on docker-dev (site-a = enablement demo, serialization removed, guard on; site-b keeps depends_on-serialization + guard off as the A/B control): simultaneous start -> site-a-1 founds, site-a-2 probes-reachable-joins -> 250/240, NO split; higher-node-alone -> probes dead founder, self-forms -> 250; founder rejoin -> 240. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
344b0d3334 |
fix(cluster): validate split topology against EFFECTIVE roles, not raw Cluster:Roles
Code review found SplitTopologyTransportValidator read the wrong role view,
defeating its fail-loud purpose two ways:
(a) Roles-source divergence. AkkaClusterOptions.Roles binds Cluster:Roles
but falls back to OTOPCUA_ROLES (RoleParser.Parse) when it is empty. A
node configured with only OTOPCUA_ROLES=driver,cluster-SITE-A genuinely
carries the cluster role in Akka, yet the validator saw no cluster role
and passed silently on MeshTransport:Mode=Dps (the documented incident).
(b) Case. The Cluster:Roles bind path is verbatim while RoleParser.Parse
lowercases the OTOPCUA_ROLES path; 'Cluster-SITE-A' / 'Driver' / 'Admin'
slipped past the ordinal checks.
Resolve effective roles the same way the node does — Cluster:Roles, else
RoleParser.Parse(env OTOPCUA_ROLES) exactly as Program.cs — then normalize
both (trim + ToLowerInvariant) before IsClusterRole/driver/admin. The message
still names the ClusterId in the operator's original spelling. Exemption and
all other behaviour unchanged. Contained to the validator.
Tests: OTOPCUA_ROLES-only cluster role on Dps fails; mixed-case roles on Dps
fail; cluster role with neither admin nor driver needs only ClusterClient;
Cluster:Roles wins over a stray OTOPCUA_ROLES. 16/16 green (124/124 project).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
|
||
|
Joseph Doherty
|
a886d5e6e0 |
feat(cluster): refuse to boot a split-topology node on a DPS transport mode
Per-cluster mesh Phase 6 (Task 5). A node carrying a cluster-{ClusterId}
role (RoleParser.IsClusterRole) is in the split topology, where the
Phase 2/3/5 DPS dark-switch branches deliver nothing across the mesh
boundary. SplitTopologyTransportValidator fails host start unless such a
node uses the mesh-crossing transports: MeshTransport:Mode=ClusterClient
always; Telemetry:Mode=Grpc if it has the driver role; TelemetryDial:Mode
=Grpc if it has the admin role. It is a no-op for any node with no
cluster-role (legacy / single-mesh / test). Roles + the three modes are
cross-read from IConfiguration, mirroring ConfigSourceOptionsValidator.
Registered via AddValidatedOptions on MeshTransportOptions with
ValidateOnStart, alongside the sibling validators.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
|
||
|
Joseph Doherty
|
a4d3ab4005 |
docs(mesh): fix stale redundancy-singleton comments + prove the driver fallback end-to-end
Review follow-ups on the redundancy re-home (Phase 6, Task 2):
- Program.cs: the hasDriver comment claimed IClusterRoleInfo "throws at host
start if this driver node carries no cluster role" — false since the fallback
landed. Rewrite it to describe the cluster-{ClusterId} scope with the driver-role
fallback for legacy single-mesh / not-yet-migrated nodes.
- ClusterRoleInfo.cs: the SubscriberActor comment described RedundancyStateActor as
the "admin-role singleton" — stale. Note it is now a cluster-{ClusterId}-scoped
singleton spawned on every driver node (LeaderChanged stays a no-op here).
- Add a host/registry-level test for the driver-role FALLBACK path (roles ["driver"],
no cluster role) closing the coverage asymmetry — it was proven only in the pure
helper. Asserts the node boots and registers RedundancyStateActorKey, i.e. the boot
the earlier throw-based version would have aborted.
The boot helper now supplies an in-memory IDbContextFactory and a fake IClusterRoleInfo:
Akka.Hosting invokes singleton props factories eagerly at StartAsync, and the admin
ClusterNodeAddressReconciler factory reads both (the latter for its per-cluster reconcile
scope, added by a sibling Phase 6 task) — without them the admin boot NREs.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
|
||
|
Joseph Doherty
|
2d2f1decae |
feat(mesh): ClusterRoleInfo exposes the node's own cluster-scoped role
Phase 6 Task 1. IClusterRoleInfo gains ClusterRole/ClusterId, derived from the node's configured AkkaClusterOptions.Roles at construction time (not live Cluster.State) so the identity is available before the cluster forms. First cluster-scoped role wins when more than one is configured, logged as a Warning. Updates the FakeClusterRoleInfo test double in ServiceCollectionExtensionsTests to satisfy the new interface members. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
87ff00fd30 |
feat(mesh): RoleParser accepts cluster-{ClusterId} roles; consolidate the driver-role literal
Phase 6 (per-cluster mesh split) needs nodes to carry a cluster-scoped role like cluster-SITE-A; RoleParser previously rejected anything outside the fixed admin/driver/dev set. Adds IsClusterRole/ ClusterIdFromRole plus well-known-role constants, and points the three duplicate "driver" string literals (RedundancyStateActor, ClusterNodeAddressReconcilerActor, ServiceCollectionExtensions) at the new RoleParser.Driver so the value has one source, without renaming any existing DriverRole symbol. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
53ae0100a2 |
feat(mesh-phase5): Telemetry/TelemetryDial options + fail-closed validators
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
a845a6d2fd |
feat(mesh-phase5): telemetry.proto contract + oneof + contract-lock test
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
3a590a0cb7 |
refactor(mesh-phase4): retire EfAlarmConditionStateStore + drop dead ScriptedAlarmState table (Task 9)
Scripted-alarm condition state lives in LocalDb (Phase 4); the ConfigDb-backed Ef store + ScriptedAlarmState table are now dead. Removes the test-harness Ef fallback (a DB-backed node with no store now skips the alarm host), deletes the store + entity + model config, and drops the table via migration. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
4f4cdd05ec |
feat(mesh-phase4): LocalDb alarm-condition-state store (replicated, pair-local)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
d630a7e267 |
feat(mesh-phase4): driver-only node must be FetchAndCache (validator)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
62e8f54f71 |
feat(mesh): deployment_artifact.v1 proto + first in-repo Grpc.Tools codegen
The repo has consumed packaged gRPC clients (LocalDb.Replication, HistorianGateway.Client) but never compiled a proto locally. This adds the Grpc.Tools toolchain to Commons and a deployment_artifact.proto with a single server-streaming Fetch(deployment_id) -> stream ArtifactChunk RPC, generating both the server base and the client stub (GrpcServices="Both") so central (Host/AdminUI) and the node (Runtime) share one reference. A compile-touch test pins the codegen: it stops compiling if generation regresses. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
9d1e60c00c |
feat(mesh): ConfigSource/ConfigServe options + validator (Phase 3 dark switch)
ConfigSource:Mode = Direct (default, read central SQL) | FetchAndCache (fetch the artifact from central over gRPC, read LocalDb). The validator fails host start on a FetchAndCache shape that cannot fetch — no endpoints, a non-http(s) endpoint, no shared key, or a non-positive timeout — because each otherwise surfaces as a silent absence (a deploy that never applies). ConfigServe (the central serve surface) takes a plain Configure: a 0 port is a valid "disabled". Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
d5b5cb6ede |
feat(mesh): CentralCommunicationActor — dark-switched fan-out, DB-sourced contacts
Phase 2 Task 3. Central-side end of the boundary: routes every central->node command out over DPS or ClusterClient per MeshTransport:Mode, and forwards inbound ApplyAcks to the deploy coordinator singleton (Forward, not Tell, so the coordinator sees the node rather than this relay). Three things here are load-bearing and easy to get wrong later: SendToAll, never Send. Today's DPS publish reaches EVERY DriverHostActor and the node side has no ClusterId or node filter to compensate -- scoping happens later, inside the artifact. ClusterClient.Send delivers to exactly ONE registered actor, so it would deploy to a single node while every other node silently kept its old config, and the deployment could still seal green. Sabotage-verified: swapping to Send reddens exactly that one test. Exactly ONE ClusterClient, fleet-wide. A receptionist serves its whole cluster, so SendToAll reaches every registered node-comm actor in the mesh regardless of which contact point was dialled. One client per application Cluster -- the shape Phase 6 wants -- would fan each command out once per cluster while the fleet is still one mesh: N x duplicate DispatchDeployment and ApplyAck. Marked TODO(Phase 6). The contact set does NOT scope delivery. Excluding a maintenance-mode node from the contacts does not stop it receiving commands; it still receives them, as it does under today's broadcast. The filter is about which receptionists are worth dialling, not about who gets the message. Stated in the code because the opposite is the natural assumption. Also carries the sister project's shipped fixes: per-row address parsing so one malformed ClusterNode cannot abort the refresh and leave the contact set half-built (regression test orders the bad row FIRST), the cache entry cleared before recreate so a failed create cannot leave commands routing into a stopping actor, and a Status.Failure handler so a DB outage is a Warning rather than a silent debug-level unhandled message. Two test-harness bugs found and fixed while writing this, both mine: SubscribeAck goes to the SENDER (so the mediator subscribe needed an explicit sender), and EventFilter matches case-INSENSITIVELY, so a "was NOT" filter also caught this actor's own "was not created" warning. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
0d6669d5ff |
feat(mesh): MeshCommand envelope + generation-suffixed ClusterClient factory
Phase 2 Task 2. Two small pieces the comm actors are built on. MeshCommand carries a command plus the DPS topic it would have been published on, so the admin singletons stop knowing which transport is in force. Without it the dark switch would have to be threaded through five publish sites across two actors, each free to drift. Central-side only -- it never crosses the wire, so it carries no serialization contract. DefaultMeshClusterClientFactory appends a generation counter to the actor name. Context.Stop is asynchronous and the name stays reserved until termination completes, so recreating the same name inside one message handler throws InvalidActorNameException -- which is exactly what a contact-set change does (stop the old client, create the replacement, one handler). Ported from the sister project, where it was a shipped bug fix rather than foresight. Sabotage-verified: freezing the name to a constant reddens both name tests with InvalidActorNameException. Dropped a third test I had written -- it asserted ClusterClientSettings directly and never touched the factory, so it was coverage theatre. Replaced with a comment saying where contact propagation IS covered (the Task 8 boundary test, the only place it can actually fail). Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
7b71a6a35d |
feat(mesh): receptionist extension, zero ClusterClient buffering, frame-size logging
Phase 2 Task 1. Three changes to the embedded base config, plus a test that reads every one of them back off a RUNNING ActorSystem rather than off the HOCON text -- the settled idiom here (SplitBrainResolverActivationTests), because several fragments compete and the losing one still reads correctly in the file. - Registers the ClusterClientReceptionist extension. It is not auto-started, so without this the boundary only listens if some code path happens to call ClusterClientReceptionist.Get(system) first: "is the boundary up?" would depend on startup ordering rather than on configuration. - buffer-size = 0. This is the one genuine behaviour change and it is NOT what the sister project runs -- they never wrote this section and inherit 1000. Buffering would replay a DispatchDeployment on reconnect and apply a deployment the coordinator already sealed as TimedOut, leaving the node on a configuration the database says failed. - log-frame-size-exceeding = 32000b. An oversized frame is dropped WITHOUT tearing down the association: heartbeats keep flowing, the node reports healthy, and the only symptom is a command that never arrived. Our deploy notify is payload-free, so this canary should stay quiet. reconnect-timeout and receptionist.role restate Akka defaults; they are stated explicitly and pinned because the behaviour is load-bearing, and the comments say which is which. One test was VACUOUS on first write: Config.GetInt returns 0 for a missing key, so the buffering assertion passed before the feature existed -- it could not distinguish "explicitly 0" from "absent, default 1000 in force". Rewritten to assert through ClusterClientSettings/ClusterReceptionistSettings, the objects the client is actually built from. Sabotage-verified after the fix: setting buffer-size back to 1000 reddens exactly that one test. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
5439f14804 |
feat(mesh): MeshTransportOptions — dark switch between DPS and ClusterClient
Phase 2 Task 0. Binds a validated `MeshTransport` section selecting which transport carries central->node commands, defaulting to Dps so nothing changes until the rig gate passes. The validator exists because every fault it catches otherwise surfaces as an ABSENCE -- a deployment that never arrives, an alarm ack that does nothing -- which has no stack trace and no failing node to point at. Two of the shapes it rejects are ported from the sister project's shipped mistakes: a contact point carrying an actor-path suffix, and (documented in the options XML) a template listing the node's OWN remoting port as a central contact, which is a permanent failure in the initial-contact rotation. Contact points are required only under ClusterClient mode; requiring them under the default would make the section mandatory on admin-only nodes that have no reason to carry them. Sabotage-verified: relaxing the empty-contacts guard and the actor-path-suffix guard turns exactly those two tests red, and nothing else. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
ee69caa270 |
feat(config): add ClusterNode.MaintenanceMode — the hatch the live gate proved missing
Phase 1 gate step 4 failed: setting Enabled = 0 to take a node out of service returns 422 ClusterEnabledNodeCountMismatch, because DraftValidator.ValidateClusterTopology requires the enabled-node count to equal ServerCluster.NodeCount. On a Warm/Hot pair — every cluster on the rig, and every cluster in the target topology — Enabled can therefore never be the maintenance hatch. The plan called step 4 "the check that makes step 3 acceptable to ship", so Task 3's behaviour change was not shippable as it stood: a node down for maintenance would block every deployment to its cluster. The two rules were each reasonable and contradictory together — the validator reads Enabled as "part of the declared topology", Phase 1 additionally read it as "expect an ack". Split the meanings rather than weaken either rule: Enabled part of the declared topology (validator, untouched) MaintenanceMode expected to participate now (coordinator + reconciler) Rejected alternatives: counting configured rather than enabled nodes (drops the guard against booting a pair into InvalidTopology); downgrading the rule to a warning (weakens a deploy gate for everyone); shipping with no hatch. Sabotage: dropping !n.MaintenanceMode from the coordinator query turns the new test red. It also asserts both nodes remain Enabled, so the fix cannot quietly regress to disabling the row after all. Configuration.Tests 95/95, ControlPlane.Tests 101/101, solution builds clean. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
da74ebd696 |
feat(config): migration AddClusterNodeTransportPorts
Additive only — two AddColumn ops, no AlterColumn, so no accumulated model drift is riding along with this change: ALTER TABLE [ClusterNode] ADD [AkkaPort] int NOT NULL DEFAULT 4053; ALTER TABLE [ClusterNode] ADD [GrpcPort] int NULL; Adds a third test covering the DB-side default specifically. The entity round-trip cannot see it: ClusterNode.AkkaPort's CLR initializer is also 4053, so that assertion passes with the mapping default deleted. The new test inserts through raw SQL with the column omitted, so only the schema can supply the value — verified by setting defaultValue: 0, which turns exactly that one test red and leaves the other two green. Configuration.Tests 95/95. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
2bbb02713c |
feat(config): add ClusterNode.AkkaPort/GrpcPort — central's dial targets
Per-cluster mesh Phase 1 groundwork. Once the meshes split (Phase 2) central can no longer see a site node's appsettings, so the transport ports it must dial have to live in a row central can read. AkkaPort is non-nullable with a 4053 default — every node listens on a remoting port, so 0 is never a truthful value and pre-existing rows must migrate to something real. GrpcPort is nullable with no default: nothing listens on it until Phase 5, and a non-null default would assert a port that does not exist. Both are documented as central's dial targets rather than the node's own binding config; the duplication against Cluster:Port is reconciled in Task 4. The new SchemaCompliance test is red until the migration lands (Task 2). Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
3f24d4d6bf |
feat(cluster)!: self-first seed ordering replaces the InitJoin self-form watchdog
Akka runs FirstSeedNodeProcess -- the only bootstrap path that can form a NEW
cluster when no peer answers InitJoin -- exclusively when seed-nodes[0] is the
node's own address. Every other node runs JoinSeedNodeProcess and retries
InitJoin forever. That is why "both peers are seeds" never meant "either can
cold-start alone", and it is fixed here the way Akka itself intends: each seed
node lists ITSELF first.
- docker-dev central-2 now lists itself as SeedNodes__0 (central-1 was already
self-first). The site nodes are untouched -- they seed off central-1 only and
are deliberately not seeds.
- AkkaClusterOptionsValidator enforces the invariant at boot via the shared
ZB.MOM.WW.Configuration AddValidatedOptions/ValidateOnStart seam. The rule is
CONDITIONAL -- self must be seed[0] only IF self is in the list at all -- or it
would refuse to boot every driver-only site node. Identity is compared on
PublicHostname (falling back to Hostname when blank) AND port, i.e. the address
Akka puts in SelfAddress: matching the 0.0.0.0 bind address would find no seed
anywhere and leave the rule silently inert on the whole docker-dev rig.
- ClusterBootstrapFallback + Cluster:SelfFormAfter are DELETED. The watchdog sat
outside Akka's join handshake, so it could not distinguish "no seed answered"
from "a seed answered and the join is in flight" -- and Cluster.Join(SelfAddress)
is not ignored mid-handshake, it wins. The live gate (
|
||
|
Joseph Doherty
|
ea45ace1c3 |
fix(cluster): don't self-form while a seed peer is reachable — the fallback islanded a restarting node
The live gate for the manual-failover control caught the self-form fallback forming a SECOND cluster. A node bounced by a failover restarted, received InitJoinAck from its live peer — the join was in flight and healthy — but did not get the Welcome inside the 10s window, because the peer's ring still held the node's previous incarnation (Exiting -> Down -> Removed). The fallback fired on the timer and the node islanded itself until an operator restarted it. The original design assumed Cluster.Join(SelfAddress) would be ignored mid-handshake. It is not — it wins. And since manual failover deliberately produces that restart, every failover could island the node it bounced. Before self-forming, TCP-probe the other seed addresses; a reachable peer means wait another window instead. That is also what the fallback claims to detect: 'no seed answered InitJoin (peer down at boot)'. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
9ded86342e |
feat(cluster): InitJoin self-form fallback armed via Akka.Hosting startup task
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
979dc102dd |
feat(cluster): SelfFormAfter option (bootstrap self-form fallback window)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
0de1352e55 |
fix(deps): pin System.Security.Cryptography.Xml 10.0.10 — fresh restores were red
Four NU1903 high-severity advisories (GHSA-23rf-6693-g89p, GHSA-8q5v-6pqq-x66h, GHSA-cvvh-rhrc-wg4q, GHSA-g8r8-53c2-pm3f) landed in the NuGet audit data against System.Security.Cryptography.Xml 10.0.7, which Microsoft.AspNetCore.DataProtection 10.0.7 pulls in transitively for key storage. Under TreatWarningsAsErrors a fresh restore produces 204 NU1903 errors and the solution does not build at all. This was invisible locally: machines with cached audit data keep building, so the break only shows on a clean clone or a docker image build. It surfaced here by accident — a detached worktree at the pre-change baseline, created to check whether a flaky test predated today's work, could not restore. Worth noting that the diagnostic path was the accident, not the intent: nothing in the normal build or test loop would have reported this before someone else hit it. Same surgical-pin pattern as the SQLitePCLRaw entry directly above it: a direct PackageReference at the one project where the chain enters the repo (Core.Configuration), rather than CentralPackageTransitivePinningEnabled, which this repo already documents as breaking the Roslyn version split. Bumping the DataProtection parent to 10.0.10 instead was rejected for the reason the sister repo recorded when it hit the same advisories: 10.0.10 floors Microsoft.Extensions.* and, via the EFCore adapter, Microsoft.EntityFrameworkCore at 10.0.10, forcing a family-wide servicing bump and an NU1605 downgrade cascade that deserves its own reviewed change. Verified the way the defect demanded — in a fresh worktree, not this one: restore clean (0 errors, down from 204) and full solution build clean. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
50b55ee4b9 |
fix(redundancy): elect the Primary by cluster age, not by address
RedundancyStateActor derived the driver Primary from ClusterState.RoleLeader("driver").
Akka offers two different notions of a "first" member and they are not the same
one: role leader is the lowest-ADDRESSED Up member (host, then port), while
ClusterSingletonManager places singletons on the OLDEST — lowest up-number.
They agree on a freshly-formed cluster, which is why every existing test passed.
They diverge after any restart: the restarted node re-joins as the youngest while
keeping its address, so if it holds the lower address it becomes role leader while
the singletons stay put. The snapshot would then name a Primary that is not
hosting the work, and every Primary-gated surface follows it — inbound device
writes, native-alarm acks, the fleet-wide alerts emit, and the alarm-history
drain would all enable on the wrong node while the node actually running the
singletons stayed gated off.
BuildSnapshot now selects the oldest Up member carrying the driver role, matching
singleton placement. Leaving members are excluded: a node handing its singletons
over must not be named Primary.
NodeRedundancyState.IsRoleLeaderForDriver is renamed IsDriverPrimary, and
NodeHealthInputs.IsDriverRoleLeader likewise. Keeping the old names would have
left the wire contract asserting a derivation the code no longer uses — the same
drift that made this defect invisible.
Proven by a real two-node cluster rather than a mock. RedundancyPrimaryElectionTests
binds the first-joining node to the HIGHER port, so oldest and lowest-address name
different nodes, and includes a fixture assertion that the divergence actually
occurred — without it the real assertion could pass for the wrong reason. Positive
control: restoring the RoleLeader derivation turns exactly the two election tests
red while the fixture check stays green.
Runtime.Tests 440 passed, ControlPlane.Tests 82, Cluster.Tests 36.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
|
||
|
Joseph Doherty
|
2964361a6f |
fix(cluster): auto-down downing strategy — a two-node pair could not survive an oldest-node crash
Ports the sister project's live-proven fix (ScadaBridge cf3bd52f). OtOpcUa ran the identical configuration it indicts: keep-oldest with down-if-alone = on. Akka.NET 1.5.62's KeepOldest.OldestDecision only lets the down-if-alone branch rescue a side holding >= 2 members. A two-node cluster losing a peer is always a 1-vs-1 split, so the branch never fires, the lone survivor falls through to DownReachable and downs ITSELF, and run-coordinated-shutdown-when-down then terminates it. down-if-alone is a 3+-node feature and does not do what its name suggests for a pair: a crash of the oldest node is a total outage, which is the exact failure the redundancy pair exists to absorb. Cluster:SplitBrainResolverStrategy now selects the provider, defaulting to auto-down: Akka's AutoDowning with auto-down-unreachable-after = 15s, so the leader among the reachable members downs the unreachable peer and a crash of either node fails over in place. keep-oldest remains available for deployments that would rather take an outage than ever run dual-active during a real partition. An unrecognised value fails the host at startup rather than falling through to the fatal default. The tests assert the EFFECTIVE configuration — they start a real host through WithOtOpcUaClusterBootstrap and read akka.cluster.downing-provider-class back off the running ActorSystem. This is not incidental. The first draft inlined the three calls the bootstrap makes instead of calling it, which pinned the test's own wiring: sabotaging production's HOCON precedence left all 36 green. The prior file had the same shape at a smaller scale, asserting only that BuildClusterOptions returned a KeepOldestOption — true, and true of a configuration that cannot fail over. Positive control: making BuildDowningHocon emit nothing for auto-down turns exactly the two effective-config guards red. Measured while verifying rather than assumed: HoconAddMode.Append also wins here, purely because it is added last, so the mode name is not the guarantee. The comment now says so instead of asserting a precedence rule that does not hold. Not yet live-drilled on OtOpcUa. The docker-dev rig is a single six-node mesh where the 1-vs-1 pathology cannot occur; the kill-the-oldest drill belongs with the per-cluster mesh work that makes every mesh exactly two nodes (design doc 6.2 / Phase 0a). Recorded as an outstanding gate in docs/Redundancy.md. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
af545efdf5 |
feat(localdb): one-time alarm-historian.db migrator
Copies the pre-consolidation store-and-forward queue into the consolidated
database on first boot, then renames the legacy file aside. Rows are in that
file precisely because the historian could not be reached, so dropping them
on upgrade would discard exactly the alarm audit trail the queue exists to
protect.
Runs last in OnReady, after every RegisterReplicated call. It is the only
thing in OnReady that writes rows, and capture is trigger-based: a migration
that ran before registration would recover the backlog locally and never
replicate a line of it, silently and permanently.
Ids are derived from the payload rather than the plan's mig-{node}-{legacyId}
scheme. Node-prefixing solves the collision the legacy AUTOINCREMENT key
would cause -- node A's row 7 and node B's row 7 are different alarms -- but
it preserves a duplication that should be collapsed instead. A warm pair's
two legacy files OVERLAP: HistorianAdapterActor default-writes while its
redundancy role is unknown, so both nodes accepted the same transitions
during every boot window. Prefixed ids would carry those duplicates into the
merged buffer forever; equal-payload ids converge them. The same property
makes a crash between commit and rename harmless under INSERT OR IGNORE.
OnReady now takes IConfiguration rather than offering an overload that skips
the migration. A wiring mistake that silently discarded a node's undelivered
alarm history is not a mistake worth making possible.
The copy is restricted to the columns the legacy table actually has. Naming
a column an older build never wrote throws "no such column", which would
discard every row in the table rather than the one field.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
|
||
|
Joseph Doherty
|
71379816e7 |
feat(localdb): alarm S&F on LocalDb with primary-gated drain (cutover)
Moves the alarm store-and-forward buffer out of its own alarm-historian.db and into the node's consolidated LocalDb, where it replicates to the redundant pair peer. A node that dies holding undelivered alarm history no longer takes it to the grave. Tasks 2 and 3 land together, as the plan anticipated. They are not separable: the gate is a constructor argument of the rewritten sink, and a commit that replicated the queue without gating the drain would be a commit in which both nodes of a pair deliver every alarm event, continuously. That drain gate is the load-bearing part of the change, and the recon explains why it is new work rather than a refinement. Exactly-once delivery across a pair is enforced today on the ENQUEUE side, by HistorianAdapterActor: only the Primary enqueues, so the Secondary's queue is empty and its ungated drain has nothing to send. Replicating the table destroys that invariant. The gate is a Func<bool> the caller supplies, because the drain runs on a timer the sink owns rather than on a mailbox, and Core.AlarmHistorian cannot reference PrimaryGatePolicy in Runtime. Runtime supplies it via a new IRedundancyRoleView singleton that DriverHostActor publishes its Primary-gate verdict to -- the same verdict the inbound-write and native-ack gates use, so there is no second notion of am-I-the-Primary to drift. Two failure modes are deliberately closed: - The view is seeded OPEN, matching the policy's own answer for an unknown role with no driver peer. A deployment that runs no redundancy never publishes to it, and defaulting closed would silently stop its alarm history forever. - A gate that throws is read as not-now, never as permission, and a closed gate reports the new HistorianDrainState.NotPrimary rather than Idle. A Secondary's rising queue is supposed to look different from a stalled drain, and if BOTH nodes report NotPrimary the pair is misconfigured and says so instead of quietly filling toward the capacity ceiling. Row ids are a hash of the payload rather than fresh GUIDs. Both adapters accept the same fanned transition in the window before the first redundancy snapshot arrives, and under last-writer-wins an equal key converges those two accepts into one row instead of duplicating them. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
8a9cb40a72 |
feat(localdb): alarm_sf_events replicated table
Adds the alarm store-and-forward buffer to the consolidated LocalDb file and registers it for replication, so a node that dies with undelivered alarm history no longer takes it to the grave. The table keeps the legacy queue's shape rather than the status column the plan sketched. An acknowledged row is deleted, not marked delivered: that needs no second sweeper to keep the table bounded, leaves the capacity semantics untouched, and the replication engine carries the delete as a tombstone so the peer drops its copy anyway -- which is what the status column was for. last_error is retained because it is the only operator-facing record of why a row was dead-lettered. The primary key is app-minted TEXT. The legacy AUTOINCREMENT RowId cannot replicate under last-writer-wins: two nodes would independently allocate rowid 7 to different alarms and silently overwrite each other. The drain index therefore orders by enqueued_at_utc rather than insertion order, with id as a tiebreak so the ordering is total. Tables are created unconditionally, independent of AlarmHistorian:Enabled. An empty registered table costs three triggers; creating it lazily would mean a node that enables the historian later writes rows before its capture triggers exist, which is exactly the silent-loss shape OnReady's ordering comment warns about. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
a27eff3298 |
feat(localdb): boot from the pair-local artifact cache when central SQL is unreachable
Hooks the Bootstrap catch that previously went straight to Stale. On a cache hit the node serves its last-known-good configuration through the outage instead of coming up with an empty address space; on a miss, behaviour is byte-for-byte what it was. The read is unkeyed because ClusterId is only derivable from an artifact you already hold or from the unreachable central DB (recon D-1). Newest pointer wins, which is correct in every real topology - a node belongs to one cluster and its peer replicates that same row. Splits PushDesiredSubscriptionsFromArtifact out of PushDesiredSubscriptions: the cache path runs precisely when the ConfigDb read cannot succeed, so re-reading the artifact there would fail by definition. RunningFromCache is surfaced on NodeDiagnosticsSnapshot (optional param, existing call sites unaffected). A node running from cache looks entirely healthy from outside - full address space, live values - but its config is frozen and no deploy can reach it. It clears only on a real apply from central. |
||
|
Joseph Doherty
|
2bae1b4c9f |
refactor(localdb): delete the dormant LiteDB LocalCache (superseded by ZB.MOM.WW.LocalDb)
Never registered in DI and dead since Phase 6.1. Keeping two unwired cache designs invites the next reader to wire the wrong one. Deletes all three test files - LiteDbConfigCacheTests.cs was missing from the plan's list and would have failed to compile (it calls new LiteDB.LiteDatabase directly). StaleConfigFlagTests lives inside ResilientConfigReaderTests.cs, so it goes too. The XML-doc reference in ILdapGroupRoleMappingService is rewritten rather than removed: it now records that no sign-in fallback exists and that reviving one means an admin-side cache on LocalDb. |
||
|
Joseph Doherty
|
8c5e2be92e |
feat(alarms): scripted condition Quality from worst-of-input tag quality (#478)
Layer 3 of #477: a scripted alarm's condition Quality now reflects the WORST quality across its input tags, mirroring the native OT semantic (#477 L2). Plumbing (quality was silently discarded twice on the live path): - VirtualTagActor.DependencyValueChanged gains Quality (defaulted Good); the DependencyMuxActor forwards the published AttributeValuePublished.Quality it already carried; ScriptedAlarmHostActor.OnDependencyChanged pushes the real quality into the engine (was hardcoded 0u/Good). Engine (Core.ScriptedAlarms): - ScriptedAlarmEngine computes worst-of-input quality each eval (skipping not-yet-published inputs, which are a readiness concern, not a quality signal) and carries it on ScriptedAlarmEvent.WorstInputStatusCode. - A real transition carries the current worst quality so ToSnapshot's full snapshot doesn't clobber quality back to Good (e.g. transition while Uncertain). - A Bad input freezes the condition (no transition), like a comms-lost native driver; a quality-bucket change with no transition emits the new EmissionKind.QualityChanged, routed to the existing #477-L2 AlarmQualityUpdate -> WriteAlarmQuality node path (quality only, no /alerts row, no historian write). ScriptedAlarmSource skips QualityChanged so it never fabricates a phantom IAlarmSource event. Host: ToSnapshot maps WorstInputStatusCode -> OpcUaQuality; OnEngineEmission routes QualityChanged out of band. Tests (TDD, RED-first): engine worst-carry + Bad/restore QualityChanged + unchanged-bucket-no-emit; source swallows QualityChanged; mux forwards quality; host Bad-dep -> AlarmQualityUpdate(no alerts) + transition snapshot carries worst. Docs: AlarmTracking.md Layer-3 section + design doc. Closes #478 |
||
|
Joseph Doherty
|
db751d12a5 |
feat(alarms): condition Quality tracks source connectivity (#477)
Part 9 ConditionType.Quality was never assigned; default(StatusCode)==Good so every native + scripted condition reported Good unconditionally — a comms-lost device still showed a healthy, inactive, Good condition (a wrong-VALUE bug, distinct from the null-value #473/#475). Clients (and HMIs bucketing on IsGood) could not tell "genuinely inactive" from "lost contact". Layer 1 — make Quality a real, plumbed field: - AlarmConditionSnapshot gains OpcUaQuality Quality (default Good). - MaterialiseAlarmCondition sets it (native BadWaitingForInitialData, scripted Good). - WriteAlarmCondition projects snapshot.Quality; the delta-gate gains a Quality member so a quality-bucket change fires a Part 9 event. Layer 2 — drive native quality from driver connectivity (a comms-lost driver emits no alarm transitions, and an alarm-bearing raw tag has no value variable, so quality can't come from either existing channel): - DriverInstanceActor Tells parent ConnectivityChanged on Connected/Reconnecting. - DriverHostActor fans it to every native condition the driver owns as OpcUaPublishActor.AlarmQualityUpdate (Good on connect, Bad on disconnect). - New dedicated IOpcUaAddressSpaceSink.WriteAlarmQuality sets ONLY Quality and fires only on a bucket change — never touches Active/Acked/Retain (an active alarm that loses comms stays active). Not a full-snapshot re-projection, so it can't clobber severity/message and works for a never-fired condition. Forwarded through DeferredAddressSpaceSink (F10b trap; auto-verified by the reflection forwarding guard). Ungated by redundancy role; no /alerts row. Scripted conditions stay Good; worst-of-input quality deferred to #478 (Layer 3). Tests: node-level (materialise/project/no-clobber/unknown-node no-op), NativeAlarmProjector, DriverInstanceActor connectivity emission, DriverHostActor fan-out, OpcUaPublishActor routing, and the wire-level guard (Condition_event_Quality_tracks_source_connectivity_on_the_wire) — RED-verified against a simulated pre-fix always-Good server. Existing DriverInstanceActor parent probes ignore the new ConnectivityChanged. Docs: docs/AlarmTracking.md §"Condition source-data Quality (#477)"; design doc docs/plans/2026-07-17-alarm-condition-quality-477-design.md. |
||
|
Joseph Doherty
|
8ebc712eff |
feat(v3-batch4-wp4): multi-notifier native alarms (single ReportEvent → raw + equipment notifiers) + teardown symmetry
Materialize each native alarm ONCE at the raw tag (ConditionId = RawPath, Raw realm); wire the single condition as an SDK event notifier of each referencing equipment's UNS folder so one ReportEvent fans to every root without re-reporting per root (which would break Server-object dedup + Part 9 ack correlation). - New sink method WireAlarmNotifiers(alarmNodeId, alarmRealm, notifierFolderNodeIds, notifierFolderRealm) on IOpcUaAddressSpaceSink, forwarded through DeferredAddressSpaceSink + SdkAddressSpaceSink + NullOpcUaAddressSpaceSink (the forwarding-trap guard); auto-covered by the DeferredSinkForwardingReflectionTests realm + forwarding guards + a hand-written forward test. - OtOpcUaNodeManager: the normative AddNotifier(isInverse) pair + idempotent EnsureFolderIsEventNotifier per equipment folder; tracked per condition in _alarmNotifierWiring. Teardown symmetry: RemoveNotifier(bidirectional:true) on RebuildAddressSpace, RemoveAlarmConditionNode, and RemoveEquipmentSubtree so no inverse-notifier entry leaks across redeploys. - AddressSpaceApplier.MaterialiseRawSubtree wires notifiers for each native alarm tag, resolving its ReferencingEquipmentPaths (Area/Line/Equipment) to the EquipmentId folder NodeIds via BuildEquipmentIdByFolderPath. - AlarmTransitionEvent gains ReferencingEquipmentPaths (empty default); /alerts renders the referencing-equipment list as display metadata. - Un-skipped + rewrote the native-alarm dark tests (DriverHostActorNativeAlarmTests x6, DriverHostActorNativeAlarmAckRoutingTests x1) for the v3 raw-condition model; new NodeManagerMultiNotifierAlarmTests proves multi-notifier wiring + teardown symmetry (no leaked duplicates after a re-trip) + applier wiring test. Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox |
||
|
Joseph Doherty
|
2e0743ad25 |
fix(v3-batch4-wp3): realm-qualified write routing + dormant discovery guard + self-correction/byte-parity tests (Wave B review H1/M1/M2/L1/L3)
H1 (HIGH): write-routing key now (AddressSpaceRealm, bareId), not bare-only. A raw s=<RawPath> and a UNS s=<Area/Line/Equip/Eff> can collide as bare strings; the bare-only key let a colliding raw+UNS pair route to the WRONG driver ref (last-writer-wins). The realm the node manager resolves (RealmOf) is now threaded through IOpcUaNodeWriteGateway.WriteAsync -> RouteNodeWrite -> _driverRefByNodeId keyed by (realm, bareId). New regression test: Colliding_raw_and_uns_bare_ids_route_to_their_own_driver_by_realm. M1 (MEDIUM): discovered-node injection made coherently DORMANT. HandleDiscoveredNodes hard-short-circuits (single enforcement point; _discoveredByDriver never populates so the re-inject tail is inert too), with a clear log pointing at the /raw browse-commit flow. New pin: Discovered_nodes_are_ignored_dormant_in_v3; the 16+2 v2 injection scenarios re-pointed to an accurate skip reason (DiscoveryInjectionDormantV3). M2 (MEDIUM): realm-qualified dual-node self-correction tests — Failed_uns_write_reverts_uns_node_and_leaves_raw_node_untouched + Raw_realm_revert_reverts_raw_node_only (the second fails if the realm is dropped). L1: removed the = AddressSpaceRealm.Uns defaults from the consequential node-manager mutation methods (WriteValue/WriteAlarmCondition/MaterialiseAlarmCondition/ EnsureFolder/EnsureVariable/UpdateFolderDisplayName/UpdateTagAttributes/ RaiseNodesAddedModelChange/Remove*/RevertOptimisticWriteIfNeeded) + the AttributeValueUpdate/AlarmStateUpdate records, so the compiler forces explicit realm; read-only accessors + internal builders retain their defaults. L3: fixed the stale VirtualTagHostActor class comment (V3NodeIds.Uns, not the retired EquipmentNodeIds.Variable). Also: DeploymentArtifactRawUnsParityTests — Raw/UNS node-set byte-parity round-trip between AddressSpaceComposer.Compose and DeploymentArtifact.ParseComposition. Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox |
||
|
Joseph Doherty
|
3efcf8014b |
feat(v3-batch4-wp3): raw-only binding + UNS fan-out + write routing
Wave B of Batch 4 — the runtime binding seam for the dual namespace. Applier (both realms, explicit realm at every sink call site): - MaterialiseRawSubtree: Raw containers as folders + Raw tags as variables keyed by RawPath (native-alarm tag → single Part 9 condition at the RawPath), all in AddressSpaceRealm.Raw; historian tagname = override else RawPath. - MaterialiseUnsReferences: each UNS reference Variable under its equipment folder (Uns realm) + an Organizes UNS->Raw edge; inherits writable/array/ historian tagname from the backing raw tag (both NodeIds -> one tagname). - FeedHistorizedRefs / ProvisionHistorizedTags now source RAW tags (mux ref stays single, keyed by RawPath); ApplyPureRemove tears down raw tags + UNS refs in place (raw-container removal falls back to rebuild). DriverHostActor (dual-NodeId, single-source fan-out): - _nodeIdByDriverRef value gains a realm (NodeRealmRef); rebuilt from RawTags UNION UnsReferenceVariables so one (DriverInstanceId, RawPath) fans to the raw NodeId AND every referencing UNS NodeId with identical value/quality/timestamp. Write inverse map keyed by the bare id; the ns-qualified NodeId the write hook passes is normalised (BareNodeId) so a write to either NodeId resolves the same driver ref (-> RawPath write). - Native raw alarm condition routing is realm-tagged (Raw); AttributeValueUpdate + AlarmStateUpdate carry the realm through to the sink. Retire EquipmentNodeIds -> V3NodeIds.Uns (applier/VirtualTagHostActor) and RawPaths.Combine (DiscoveredNodeMapper, discovered nodes are Raw now). DeploymentArtifact.ParseComposition emits the Raw + UNS subtrees byte-parity with the composer (reconstruct entities -> AddressSpaceComposer.Compose). Sink surface: removed the transitional `= AddressSpaceRealm.Uns` defaults from IOpcUaAddressSpaceSink / ISurgicalAddressSpaceSink / SdkAddressSpaceSink / DeferredAddressSpaceSink / NullOpcUaAddressSpaceSink — every call site is now explicit (realm reordered before the trailing optionals on EnsureVariable + MaterialiseAlarmCondition). Node-manager convenience methods keep their defaults (they are not the interface impl; Sdk delegates explicitly). Tests: rewrote DriverHostActorLiveValueTests (fan-out drift, 1:N) + DriverHostActorWriteRoutingTests (dual-NodeId raw/uns write routing) to the v3 raw+uns model; new AddressSpaceApplierRawUnsTests; migrated the EquipmentTags provisioning/feed tests to RawTags; swept EquipmentNodeIds test callers. Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox |
||
|
Joseph Doherty
|
e95615cef3 |
fix(v3-batch4-wp2): AddReference sink seam for Organizes UNS→Raw (Wave A review M1)
Close the Wave-A M1 gap: the sink surface had no way to wire the mandated cross-tree Organizes reference from each UNS reference Variable to its backing Raw node, forcing WP3 to reopen the frozen surface. Add a dedicated realm-qualified AddReference method instead. - IOpcUaAddressSpaceSink.AddReference(sourceNodeId, sourceRealm, targetNodeId, targetRealm, referenceType="Organizes"): realm-qualified both ends; idempotent; a missing endpoint is a logged no-op (never throws) so a mid-rebuild race can't fault a deploy. Base-interface capability (no surgical sniff, no transitional default — WP3 wires it explicitly per UNS reference variable). - SdkAddressSpaceSink forwards to the node manager; DeferredAddressSpaceSink forwards unconditionally (like EnsureFolder); NullOpcUaAddressSpaceSink no-ops. - OtOpcUaNodeManager.AddReference: resolve both nodes by full ns-qualified key (variable/folder/condition via ResolveNodeState), guard both-exist, then wire the edge bidirectionally (forward Organizes on source, inverse on target) with a ReferenceExists idempotency guard + ClearChangeMasks on mutated sides. Reference type resolved by ResolveReferenceType (Organizes default). Added internal GetNodeReferences test/diagnostic accessor. - Reflection guard: the exhaustive-forwarding test + the realm-discriminator guard auto-cover AddReference (it has two AddressSpaceRealm params) — no edit needed. - New SdkAddressSpaceSinkTests: Organizes UNS→Raw edge created bidirectionally + idempotent; missing endpoint is a safe no-op. - All 15 IOpcUaAddressSpaceSink test doubles gain the AddReference no-op so the solution still builds. Build: dotnet build ZB.MOM.WW.OtOpcUa.slnx = 0 errors. Tests: Commons.Tests 310/310; OpcUaServer.Tests 337 passed / 4 pre-existing skips. Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox |
||
|
Joseph Doherty
|
5534698d70 |
feat(v3-batch4-wp2): node manager dual-namespace + realm-aware sink surface
Register both v3 namespaces (Raw first, UNS second) on OtOpcUaNodeManager and thread an AddressSpaceRealm discriminator through every node-naming sink method so a bare node id is resolved to the correct namespace by realm — never parsed out of the id string. - IOpcUaAddressSpaceSink / ISurgicalAddressSpaceSink: every node-naming method gains `AddressSpaceRealm realm = AddressSpaceRealm.Uns` (transitional default so un-migrated WP3 call sites still compile; WP3/Wave B makes them explicit and removes the default). Null + SdkAddressSpaceSink impls updated; DeferredAddress- SpaceSink forwards realm through every method (the forwarding trap). - DeferredSinkForwardingReflectionTests: existing exhaustive-forwarding guard auto-covers the new signatures; added an explicit guard that every node-naming sink method carries an AddressSpaceRealm parameter (RebuildAddressSpace exempt). - OtOpcUaNodeManager: register RawNamespaceUri + UnsNamespaceUri; realm->namespace index via NamespaceIndexForRealm; all node maps (_variables/_folders/ _alarmConditions/_nativeAlarmNodeIds/_historizedTagnames/_eventNotifierSources) re-keyed by the full ns-qualified NodeId string so Raw and UNS nodes sharing a bare id stay distinct; _notifierFolders already NodeId-keyed. A historized UNS reference node registers the SAME historian tagname as its backing raw node (both NodeIds -> one tagname). Inbound-write hook routes the full ns-qualified NodeId to the write gateway (realm-aware) and reverts by bare id + realm. HistoryRead seams resolve via NodeId directly. DefaultNamespaceUri kept as a transitional alias to UnsNamespaceUri for the SubscriptionSurvivalTests. - Test doubles across Commons.Tests / OpcUaServer.Tests / Runtime.Tests updated to the new interface signatures; SdkAddressSpaceSinkTests asserts the UNS namespace index for default-realm nodes. Build: dotnet build ZB.MOM.WW.OtOpcUa.slnx = 0 errors. Tests: Commons.Tests 310/310; OpcUaServer.Tests 335 passed / 4 pre-existing skips. Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox |
||
|
Joseph Doherty
|
a1a56e22bb |
feat(v3-batch4): contracts — dual-namespace URIs, V3NodeIds, AddressSpaceRealm
Batch 4 contracts commit (coordinator lands before Wave A fan-out). Purely additive so wave agents branch from a green base: - AddressSpaceRealm enum (Raw | Uns) — the explicit realm discriminator that travels alongside a node's s= identifier at every sink seam (WP2 adds it to IOpcUaAddressSpaceSink / ISurgicalAddressSpaceSink; WP3 threads it). - V3NodeIds — the two namespace URI constants (https://zb.com/otopcua/raw, https://zb.com/otopcua/uns, replacing the single .../ns) + Raw() pass-through (s= id == RawPath) + Uns()/UnsChild() slash-joined Area/Line/Equipment[/Eff] builders, both sharing RawPaths.Separator + ordinal segment validation. Placement note: URIs live in V3NodeIds (Commons) rather than beside the retired OtOpcUaNodeManager.DefaultNamespaceUri so runtime + tests reference them without depending on OpcUaServer (single-source-of-truth, §5). WP2 rewires the node manager to register both via V3NodeIds.RawNamespaceUri/UnsNamespaceUri. EquipmentNodeIds retirement is NOT in this commit: its callers are the applier + runtime (DriverHostActor/VirtualTagHostActor/DiscoveredNodeMapper), all WP3-owned (the composer does not use it) — deleting it here would red the base. WP3 sweeps it in Wave B. Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox |
||
|
Joseph Doherty
|
dc0d7653b9 |
feat(v3-batch3): B3-WP4 — {{equip}}/<RefName> reference-relative resolution
Replace the deleted equipment-base-prefix {{equip}} mechanism (impossible now
that equipment is reference-only) with per-reference resolution:
{{equip}}/<RefName> resolves through the owning equipment's UnsTagReference rows
(by effective name) to the backing raw tag's RawPath. Every unresolved <RefName>
is a clear deploy error + authoring rejection + Monaco diagnostic — preserving
"editor accepts <=> publish accepts".
Commons:
- EquipmentScriptPaths: delete DeriveEquipmentBase; SubstituteEquipmentToken now
takes the equipment's reference map (effectiveName -> RawPath) and substitutes
{{equip}}/<RefName> inside path literals (unresolved left intact, never throws);
add ExtractEquipReferenceNames (path-literal scoped) +
ExtractEquipReferenceNamesFromText (message templates). Slash syntax replaces
the v2 dot joint.
- New EquipmentReferenceMap: shared, input-shape-agnostic builder (entity + JSON
sides) over RawPathResolver — the single authority both compose seams + the
validator use, so resolved RawPaths agree byte-for-byte.
Compose seams (byte-parity kept):
- AddressSpaceComposer.Compose + DeploymentArtifact.ParseComposition both build
the per-equipment reference map from UnsTagReferences + Tags + raw topology and
substitute VirtualTag AND ScriptedAlarm-predicate sources before dependency
extraction (runtime dep refs = resolved RawPaths).
Deploy gate + authoring + editor:
- DraftValidator.ValidateEquipReferenceResolution: EquipReferenceUnresolved error
for unresolved {{equip}}/<RefName> in VirtualTag/ScriptedAlarm scripts + alarm
message-template tokens (naming script, equipment, missing ref).
- UnsTreeService.ValidateEquipTokenAsync (Wave-A M1): reference-relative authoring
rejection, replacing the stale DeriveEquipmentBase(empty)->reject-all logic.
- ScriptAnalysisService: {{equip}}/ completion offers the equipment's reference
effective names; DiagnoseAsync flags unresolved refs (OTSCRIPT_EQUIPREF) same as
the deploy gate. Requests carry optional EquipmentId (threaded MonacoEditor ->
monaco-init.js -> fetch bodies). IScriptTagCatalog.GetEquipmentRelativeLeavesAsync
repurposed to GetEquipmentReferenceNamesAsync(equipmentId, filter).
Tests: EquipmentScriptPaths substitution/extraction (slash pinned); composer<->
artifact reference-resolution byte-parity; DraftValidator unresolved-ref;
ScriptAnalysis completion+diagnostic; M1 authoring gate. Build clean (0/0); all
five affected suites green.
Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox
|