b815446940f4c859deae66f9dfdd9c8bf00b42a8
285 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Joseph Doherty
|
b815446940 |
refactor(drivers): delete GetMemoryFootprint / FlushOptionalCachesAsync (#525)
Both `IDriver` members had no production call site. Their only consumer was `MemoryTracking`, deleted with the driver-tier recycle machinery in #522, and `GetMemoryFootprint`'s doc-comment claimed "polled every 30s by Core" — which had not been true since v3, even before #522 removed the poller. Owner decision on #525 was (a) delete rather than (b) build the process-wide watchdog that would consume them. The driver-side implementations are in git history and were mostly reusable; what was always missing is the trigger, and a member nobody calls is how the misleading doc-comment survived this long. Removed from `IDriver`, all 12 driver implementations, and every test stub — plus the support code that existed only to feed them, which is the part a member-only deletion would have left behind as dead weight: - MTConnect's three per-entry byte-weight constants, its cached `ProbeModelDataItemCount`, and `CountDataItems` — a full device-tree walk run on every cache fill to populate a number with no remaining reader. - OpcUaClient's `_discoveredNodeCount` field and the discovery-pass write to it. - MQTT's `ApproxBytesPerAuthoredTag`. Two behavioural notes, neither a change in what the software does: - `AbCipTemplateCache` now has no production reset path. Its only caller was AB CIP's flush, which itself had no caller — so nothing changes except how visible the fact is. Recorded on the type, along with the correction that its comment's claim about a device `ReinitializeAsync` also flushing it was never true. A UDT redefined on a live controller keeps its stale shape until the driver instance is rebuilt. - MTConnect's `/probe` cache is still dropped, by `InvalidateProbeModel()` on an Agent restart. That is now `internal` rather than `private` so the tests which used the flush purely to empty the cache drive the real path instead — they now cover a case that actually occurs in production. Several unrelated doc comments cited `FlushOptionalCachesAsync` as the canonical CAS example and are repointed to it. Tests: the ones that existed only to assert on the removed members are deleted (3 OpcUaClient, 3 Galaxy, 2 MQTT, 1 MTConnect, 1 AbCip). Worth noting one was vacuous — `FlushOptionalCachesAsync_resets_footprint_counter` asserted a footprint of 0 before the flush and 0 after with no discovery in between, so it would have passed against a flush that did nothing at all. Full suite green except 7 failures verified identical on master before and after (4 AbLegacy integration with the compose profile down, 3 Host.IntegrationTests), checked by name rather than assumed. Docs corrected where they described this as live: `DriverLifecycle.md` (which carried the #525 placeholder), `drivers/AbCip.md`, `drivers/AbLegacy.md`, and `v2/driver-stability.md` — whose "polls this every 30 s" + per-driver `DriverConfig` memory budget were never built as written and are now banner-ed as design intent rather than description. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
7f2108dcf2 |
feat(mqtt): Sparkplug primary-host STATE publishing, Primary-gated (#511)
`ActAsPrimaryHost` was accepted by config and by the AdminUI form and did nothing but
log a warning. It now delivers the whole three-part contract or none of it: a retained
`ONLINE` on `spBv1.0/STATE/{hostId}`, a matching retained `OFFLINE` armed as the MQTT
Last Will, and an explicit `OFFLINE` on a clean stand-down.
**The claim is a dedicated connection, not a message.** A Will can only be registered in
the CONNECT packet — it cannot be added to a live session, changed, or withdrawn. So a
claim that is acquired and released at runtime cannot ride the long-lived ingest session.
`SparkplugPrimaryHost` opens its own connection (client id `{clientId}-spbhost`, forced
clean-session) when the node gains Primary and closes it when the node loses Primary; the
connection's lifetime *is* the claim, so every leg follows from its lifecycle. Ingest is
untouched by a role change — no dropped subscriptions, no lost birth cache, no data gap at
the Secondary->Primary transition.
**Primary-gated** (owner decision, recorded on the issue). Both pair nodes subscribe
independently, so an ungated claim means both claim the same host id — and the sharp
failure is not the two fighting over the retained STATE, it is that the SECONDARY's death
fires its Will and publishes OFFLINE while the Primary is healthy, stopping every edge node
in the group. Registering the Will only on whichever node is Primary at connect time fails
the other way: a node that later gains Primary would publish ONLINE with no Will behind it.
New seam: `IDriverPrimaryRoleSource` (`Core.Abstractions`) carrying the *data-plane* verdict
(`PrimaryGatePolicy.ShouldServiceAsPrimary`), not the alarm-drain verdict — an unknown role
with a driver peer present must deny. Registered unconditionally by `AddOtOpcUaRuntime`,
deliberately NOT alongside `IRedundancyRoleView`, which `AddAlarmHistorian` registers only
when that section is enabled: piggybacking would leave a pair with no alarm historian
falling back to "claim unconditionally". Polled on a 1s timer rather than evented — a
process singleton raising events at per-instance drivers outlives every driver it holds.
Accepted trade: a brief edge-visible OFFLINE window during failover. That is a genuine
state transition, not a glitch to hide.
Live-gated 5/5 against the Mosquitto fixture, including the Will leg — `AbandonForTests`
drops the socket with no DISCONNECT, because a clean disconnect makes the broker discard
the Will by protocol rule, so a test driven through `StopAsync` would observe the
stand-down OFFLINE and pass against an implementation that never armed a Will. Falsified:
removing the Will turns that leg and only that leg red.
Also corrects two stale claims found en route: the AdminUI form still said MQTT tags are
read-only (#508 shipped), and the driver doc still said nothing consumes `IRediscoverable`.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
|
||
|
Joseph Doherty
|
2d20aeb779 |
fix(opcua,drivers): carry driver status sub-codes to clients (#519) + health-write ordering (#517)
#519 — a driver's OPC UA status code now reaches a client with its sub-code intact. DriverInstanceActor projected the driver's uint onto a 3-state Good/Uncertain/Bad enum and OtOpcUaNodeManager re-expanded Bad to a bare 0x80000000, so BadNoCommunication, BadTypeMismatch and BadNotSupported were indistinguishable on the wire and every driver's status constants (incl. the 16 #497 had just corrected) were diagnostic in logs only. Resolved by option (b): the raw uint rides AttributeValuePublished -> AttributeValueUpdate -> IOpcUaAddressSpaceSink.WriteValue, OpcUaQuality is deleted, and severity classification is a helper (OpcUaStatus.IsGood/IsUncertain/IsBad). The surface was wider than the issue costed (41 files, not "the sink and 3 implementations"): OpcUaQuality had non-driver producers with no uint to carry. Each got its natural code — BadInternalError for a failed virtual-tag script, BadNoCommunication for driver comms loss — and ScriptedAlarmHostActor's pair of mutually-inverse uint<->enum mappers is deleted, having round-tripped a code the engine already held. AddressSpaceApplier's removal writes deliberately keep the generic Bad: not losing a code the driver had is this issue; inventing one for the deploy path is a separate call. Behaviour change: WriteAlarmQuality fires its Part 9 event on a status-code change rather than a severity-bucket change. While the annotation was 3-state those were the same predicate; now a move between two Bad reasons changes what a subscriber reads. New SDK-free named constants live on Commons.OpcUa.OpcUaStatus, and StatusCodeParityTests is widened to cover them — that guard is reflection-driven, so it can only see a named constant, never an inline literal. #517 — the symptom is real but the issue named the wrong path and its fix was unsafe. ShutdownAsync cannot race: DriverInstanceActor calls it from PostStop, after the mailbox drains. (Core/Hosting/DriverHost, the one class with a concurrent GetHealth, has no production call site.) InitializeAsync can: it runs on a fire-and-forget Task.Run, and MTConnectDriver writes Healthy then starts loops that can throw into a catch which disposes the client before writing Faulted. Moving the write earlier, as suggested, inverts the terminal state — teardown drains in-flight loops that write health as they unwind, so a late loop could leave Healthy at rest with every connection closed. The fix is a bracket (conservative before, terminal after), applied to MTConnect's init-failure path only; shutdown keeps its single trailing write, and the rule plus the reachability table are documented on IDriver.ShutdownAsync. CalculationDriver.GetHealth returned a hard-coded Healthy that consulted nothing — Healthy before initialize, after shutdown and after dispose. Now derived from _running. Its test, named Health_is_always_Connected, asserted the defect and is replaced. Every new guard was falsified by breaking the production code and watching it go red. The two sub-code tests each assert two different Bad codes, since a single-code assertion could pass against a hard-coded constant under the old projection. Pre-existing failures, verified unchanged on a stashed tree and not chased: 4 in Driver.AbLegacy.IntegrationTests (slc500/micrologix/plc5 fixtures not up) and 3 in Host.IntegrationTests. Closes #517 Closes #519 Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
d979ac69eb |
refactor(config): drop the dead DriverInstanceResilienceStatus table (#524)
The sibling of #521's DriverHostStatus, dead for exactly the same reason. Before this change the only reference in the entire repo was the DriverInstanceResilienceStatuses DbSet declaration: no writer, no reader, and the table empty on every deployment. Its entity doc described two components that were never built — a HostedService sampling the tracker every ~5 s into this table, and an AdminUI that left-joined it on DriverHostStatus for display. There is no sampler, that join exists in no razor file, and DriverHostStatus itself was removed in #521. Per-cluster mesh Phase 4 also makes the design unbuildable now: AddOtOpcUaConfigDb is gated on the admin role, so a driver-only node has no ConfigDb connection to write rows with. Any per-node runtime-status table has this problem. The live data was never missing — it just does not go through SQL. Resilience counters reach the AdminUI's DriverStatusPanel as DriverResilienceStatusChanged over the driver-resilience-status telemetry channel into the in-memory IDriverResilienceStatusStore. Verified consumed: the panel injects the store, fed by both DriverResilienceStatusBridge (DPS) and TelemetryDialSupervisor (gRPC). If breaker HISTORY across a central restart is ever wanted — the one thing the last-value-only store cannot give — it needs a CENTRAL-side writer fed by that stream, never a driver-node writer. Keeping an empty table now bought nothing toward that. Migration DropDriverInstanceResilienceStatusTable; the scaffolder's data-loss warning cannot bite on a table nothing ever inserted into, and Down() recreates it exactly. Also corrected: DriverResilienceStatusTracker's doc-comment claimed CurrentInFlight fed DriverInstanceResilienceStatus.CurrentBulkheadDepth "until the full telemetry observer lands". The observer landed in Phase 5; the table never had a writer. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
cbb882293b |
refactor(drivers): delete the inert driver-tier recycle machinery (#522)
The tier system was documented, operator-authorable, and inert. Deleted rather than activated,
because its premise is gone rather than merely unused.
"Tier C" meant a driver running out-of-process behind an IDriverSupervisor that could restart its
Host without tearing down the OPC UA session. No such process exists anywhere: Galaxy reaches
MXAccess over gRPC to the external mxaccessgw sidecar (PR 7.2 retired the in-process
Galaxy.Host/Proxy/Shared projects) and FOCAS has run in-process since its managed wire client
landed 2026-04-24. Consistently, IDriverSupervisor had ZERO implementations and there was nothing
for one to implement against.
The issue understated the inertness. It says the Tier-C-only protections never engaged, implying
the Tier A/B parts did. They did not: nothing constructs MemoryTracking, MemoryRecycle or
ScheduledRecycleScheduler outside their own unit tests, so the whole Core/Stability recycle layer
was dead — meaning option (a), "pass real tiers", was never a flag flip. It would have meant
writing the wiring that never existed AND arming it.
Deleted: MemoryTracking, MemoryRecycle, ScheduledRecycleScheduler, IDriverSupervisor, the
vestigial DriverTypeRegistry (referenced only by its own tests), and the RecycleIntervalSeconds
knob — which the AdminUI let an operator author and the parser validated while it configured
nothing.
DriverTier itself SURVIVES and is load-bearing: DriverResilienceOptions.GetTierDefaults supplies
the real per-capability timeout/retry/breaker policies via DriverFactoryRegistry.GetTier and
DriverCapabilityInvokerFactory. Only the isolation-and-recycle layer above it is gone.
Deliberately NOT deleted:
- WedgeDetector came along in the same directory and is equally dead in production, but it is
tier-agnostic and is not recycle machinery — it only shares the folder. Restored rather than
swept up in a decision that was not about it.
- IDriver.GetMemoryFootprint() and FlushOptionalCachesAsync() lose their only consumer here.
Removing them touches all 12 drivers and every test stub, so they are documented as
consumerless and filed as #525 instead of buried in this diff.
Compatibility: a deployed ResilienceConfig blob still carrying "recycleIntervalSeconds" parses
cleanly (unknown keys are ignored — guarded by a new test, because a blob that suddenly failed to
parse would fall back to tier defaults and silently discard the operator's real overrides), and
the AdminUI's preserve-unknown-keys bag keeps the key rather than rewriting stored config on an
unrelated edit.
The 01/U-6 knob-inertness guard carried an explicit carve-out admitting RecycleIntervalSeconds was
dormant and out of scope; that carve-out is now gone, so the expected set is literally what the
test's name claims.
Note: Host.IntegrationTests has 2 failures (DriverProbeRegistrationTests.is_idempotent,
PrimaryGateFailoverTests) — verified pre-existing by reproducing both on clean master
|
||
|
Joseph Doherty
|
30d0697c28 |
feat(hosts): surface per-host connectivity on /hosts; drop the unwritable table (#521)
`IHostConnectivityProbe` was a dead surface: eleven drivers implement it, `GetHostStatuses()` had ZERO production call sites, and `OnHostStatusChanged` had no subscriber outside the Galaxy driver's own aggregator. Per-host connectivity was computed by every driver and read by nobody. The issue offered "build the publisher or delete it". The publisher as its entity doc described it — driver nodes upserting `DriverHostStatus` rows — is not buildable: per-cluster mesh Phase 4 gates `AddOtOpcUaConfigDb` on the `admin` role, so a driver-only node has no ConfigDb connection to write rows with. So the capability is kept and the transport changed. `DriverHealthChanged.HostStatuses` now carries the probe result to `/hosts` as a Hosts column. That channel already reached the page, already survives the mesh split via the Phase 5 gRPC telemetry stream, and already replays a last-value snapshot on re-subscribe — so per-host state re-primes after a reconnect without a durable store. Both halves of the interface finally do what they are for: the event triggers a prompt publish, the pull is the source of truth. The point of the column is the case the driver-level state chip structurally cannot express: a multi-device driver stays aggregate-Healthy while ONE of its devices is unreachable. Two traps, both pinned by tests that were falsified against the prod code: - The host digest MUST be in the publish fingerprint. On a single-host-down transition every other fingerprint component is unchanged, so the dedup would swallow exactly the publish carrying the news — the trap that already bit the rediscovery signal. Removing it turns the guard test red, verified. - null (no probe) must stay distinct from empty (probe with no hosts). proto3 cannot tell an absent repeated field from an empty one, hence the explicit `has_host_statuses` flag; collapsing them would render every probe-less driver as one whose devices are all fine. Dropped: the DriverHostStatus entity, enum, DbSet, model config and table (migration DropDriverHostStatusTable — empty on every deployment, so the scaffolder's data-loss warning is moot, and Down() recreates it exactly). Found en route, NOT fixed here: `DriverInstanceResilienceStatus` is the identical defect — no writer, no reader, only a DbSet declaration, while the live data rides the `driver-resilience-status` telemetry channel. Its doc-comment now states that rather than describing the sampler and AdminUI join that were never built. Filed as #524 rather than widening this schema change beyond what was asked. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
5fff597324 |
revert(drivers): remove the periodic 5-minute device re-browse (#523)
Removes the tag-set drift detector shipped in |
||
|
Joseph Doherty
|
e77c8a3569 |
feat(drivers): tag-set drift detection, gated on SupportsOnlineDiscovery
The second thing #516/§8.2 deliberately did not build. The original objection stands — Modbus, S7, MQTT, AbLegacy and Sql echo authored config from DiscoverAsync, so diffing discovered against authored is a tautology for them — but it is now ENCODED rather than used as a reason not to build. The discriminator already existed: ITagDiscovery.SupportsOnlineDiscovery is documented as "enumerates the tag set from the live backend rather than replaying pre-declared/authored tags", and it separates the fleet cleanly — FOCAS, TwinCAT, MTConnect and AbCip true; the five config-echo drivers explicitly false. The check runs only for a driver that is SupportsOnlineDiscovery AND reports the new AuthoredDiscoveryRefs. That is nullable rather than empty-by-default on purpose: an empty collection legitimately means "nothing authored, everything discovered is new", so conflating the two would report total drift for a driver that never opted in. Where the value is: AbCip and FOCAS browse a live backend but implement no IRediscoverable, so before this they had no change signal at all. A periodic compare is the only way to notice a PLC re-download. A finding that changed the design: FOCAS, TwinCAT and AbCip all re-emit their authored tags into the same DiscoverAsync stream as device-derived ones, so the browse picker can show an operator their existing tags. Discovered is therefore always a superset of authored, and a VANISHED tag can never appear missing — one whole direction is structurally undetectable for three of the four. Shipping a Vanished list that is permanently empty for them would have been the half-inert seam this whole exercise removes. So the driver declares DiscoveryStreamIncludesAuthoredTags, the detector does not compute the undetectable half, and the operator message says "missing-tag detection unavailable for this driver" rather than letting the absence of a missing-tag clause read as reassurance. MTConnect is the only driver where both directions work — its stream is purely probe-model-derived. Behaviour: a failed browse is not drift (an unreachable device would otherwise report every authored tag as vanished); a truncated capture is skipped for the same reason; an unchanged drift is reported once rather than re-stamping its timestamp; drift that resolves clears the prompt. Interval defaults to 5 minutes — it browses a real device, and a tag set changing is an engineering event, not a runtime one. It reuses the Stage-2 surfacing, raising the same /hosts re-browse prompt, and never rebuilds the served address space. Comparison logic is a pure, separately-tested type rather than actor-inline. 14 new tests. The gate was verified load-bearing by deleting the SupportsOnlineDiscovery half: the tautology-driver test goes red, and it asserts discovery was never invoked rather than merely "no prompt raised", which would have passed for the wrong reason if the timer never fired. Full suite green except the same three pre-existing fixture-gated integration suites, identical counts. |
||
|
Joseph Doherty
|
6a01358b9a |
fix(adminui): guard the driver dispatch maps and close G-1..G-6 (§8.4)
A parity guard already existed for DriverTypeNames <-> the /raw driver picker, but not for the two dispatch maps downstream, so a driver could be registered, offered in the picker, and then have no config form. That is exactly what happened to Calculation (G-1) and to Sql/MTConnect/Calculation on the device modal (G-2) — both survived review because nothing could see them. The maps had to become data first. A Razor @switch compiles into BuildRenderTree's IL, so no test can enumerate its cases; the picker guard works only because RawDriverTypeDialog keeps its data in a field the markup enumerates. Both modals now render from DriverConfigFormMap / DeviceFormMap through <DynamicComponent>, and being public those maps need none of the picker test's BindingFlags.NonPublic fragility. - G-1 CalculationDriverForm.razor — RunTimeout was unauthorable via the UI. - G-2 Sql/MTConnect/Calculation declared single-connection rather than given hollow device forms; each holds one connection at the driver level. - G-3 DriverConfigModal's hardcoded "Galaxy or Mqtt" -> IsSingleConnection, so Sql/MTConnect authors are no longer sent to a device form with no endpoint. - G-4 DriverFormMapParityTests (5, both directions) + DriverDispatchMapParityTests. - G-5 CsvColumnMap entries for Sql, Mqtt, MTConnect. - G-6 RawBrowseCommitMapper Sql branch — and the browser end, which the audit missed: SqlBrowseSession emitted NO AddressFields, so a browsed leaf carried only a column name, and a column alone cannot address a Sql tag. It now travels schema/table/column and the mapper builds a WideRow config from them. A branch alone would have produced a half-built blob. Two findings while writing the guards. The G-6 test showed Modbus and Calculation also hit the generic address fallback — correctly, as neither is browsable — so it asserts the fall-through set EQUALS a documented non-browsable list in both directions; a one-way check would let a newly browsable driver be quietly added to the exclusion list. And TagConfigDriverTypeNameGuardTests was hollow: it enumerated a hand-written TheoryData that had already drifted (omitting Galaxy, Sql and Mqtt), guarding renames but not gaps. Now enumerated from the map with a coverage test facing the other way. Live-verified on docker-dev, since this repo has no bUnit and no unit test reaches Blazor parameter binding: opened Calculation's config (previously "No typed config form"), typed 3500, saved, reopened — the value persisted, so the DynamicComponent two-way binding round-trips. Sql's form renders fully and now reads "This driver holds a single connection, authored above". AdminUI.Tests 930 passed. |
||
|
Joseph Doherty
|
09a401b881 |
feat(drivers): consume IRediscoverable as an operator re-browse prompt (§8.2, #518)
Nine drivers raise IRediscoverable.OnRediscoveryNeeded when they observe that a remote's tag set may have changed — a Galaxy redeploy, a TwinCAT symbol-version bump, an MTConnect agent restart, a Sparkplug rebirth. Nothing in src/ subscribed, so every raise went into the void. Drivers even wrap the invoke in try/catch for "subscriber threw"; there has never been a subscriber. DriverInstanceActor now attaches the event in PreStart and detaches in PostStop, mirroring AttachAlarmSource/DetachAlarmSource. Attach is per-actor, not per-connect: the raise has no connection affinity and can land while the driver is between connects. The detach is load-bearing rather than tidy — the IDriver instance outlives the actor when the host respawns a child around the same driver object, so a missing -= accumulates one handler per respawn, each holding a dead Self. The signal rides the existing driver-health snapshot to /hosts as a "re-browse" chip. It is ADVISORY: the served address space is deliberately not rebuilt, because v3 authors raw tags explicitly through the /raw browse-commit flow and a runtime graft would materialise nodes no operator approved and no deployment artifact records. Two things worth knowing: - PublishHealthSnapshot dedups on a health fingerprint, and a rediscovery raise on an otherwise-healthy driver changes none of the four fields it hashed. Without adding the timestamp to that tuple the dedup swallows the exact publish carrying the signal. Reverting that one line turns 3 of the 5 new tests red, which is how it was confirmed rather than assumed. - The new test stub implements IDriver from scratch instead of deriving from the shared StubDriver, whose GetHealth() returns DateTime.UtcNow — its fingerprint differs on every call, so the dedup never engages and the dedup test would have passed vacuously either way. The planned discovery-result drift detector was NOT built. Modbus, S7, MQTT, AbLegacy and Sql all echo authored config from DiscoverAsync rather than browsing the device, so a discovered-vs-authored diff is permanently empty for them — it would have been another plausible-looking inert seam, which is the class this audit exists to remove. IRediscoverable is the trustworthy signal because the driver asserts it deliberately. DriverHealthChanged, IDriverHealthPublisher.Publish and HostsDriverRow gain defaulted optional parameters, so no existing call site changed. telemetry.proto gains fields 8/9 and both Phase-5 mappers carry them. Runtime.Tests 512 passed / 31 skipped. |
||
|
Joseph Doherty
|
a107a1a8b9 |
feat(mtconnect): host registration + DriverTypeNames const + guard parity (Task 16)
Register the MTConnect factory + probe with the Host, add the DriverTypeNames.MTConnect constant, and keep DriverTypeNamesGuardTests green. - DriverTypeNames: new MTConnect const + appended to All. - DriverFactoryBootstrap: MTConnectDriverFactoryExtensions.Register in Register(...) at the default Tier A (managed HttpClient/XML, in-process; Tier C is the only tier arming process-recycle, wrong here), and TryAddEnumerable(IDriverProbe -> MTConnectDriverProbe) in AddOtOpcUaDriverProbes so the probe reaches admin-only nodes (the admin-pinned Test-Connect singleton) without double-registering on a fused admin,driver node. - Host.csproj: ProjectReference to Driver.MTConnect (required to compile the Register call). - Core.Abstractions.Tests.csproj: ProjectReference to Driver.MTConnect so the guard's bin scan discovers the factory. This and the constant MUST land together -- verified RED (2/4 facts) with the constant alone. - DriverProbeRegistrationTests: MTConnect added to AdminUiDriverTypeKeys; its idempotency fact asserts distinct-probe-count and goes RED without this (and RED if TryAddEnumerable is downgraded to AddSingleton). Reconciles all four "MTConnect" literals onto the one constant: the factory's DriverTypeName, MTConnectDriver.DriverType, and MTConnectDriverProbe.DriverType now all alias DriverTypeNames.MTConnect (no circular reference -- Core.Abstractions is a leaf both driver projects already reference). This is the repo's TwinCat/Focas anti-drift convention. |
||
|
Joseph Doherty
|
c3a2b0f773 |
Merge branch 'feat/mqtt-sparkplug-driver'
# Conflicts: # src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/DriverTypeNames.cs # src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Shared/Drivers/DriverConfigModal.razor # src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/EndpointRouteBuilderExtensions.cs # src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Uns/TagEditors/TagConfigEditorMap.cs # src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Uns/TagEditors/TagConfigValidator.cs # src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/ZB.MOM.WW.OtOpcUa.AdminUI.csproj # src/Server/ZB.MOM.WW.OtOpcUa.Host/Drivers/DriverFactoryBootstrap.cs |
||
|
Joseph Doherty
|
2193d9f2e4 |
fix(config): refuse to store a Sql credential in a node config override (#499)
`ClusterNode.DriverConfigOverridesJson` is the third surface a Sql driver's config is persisted on, and the only one nothing gated. It is a map keyed by `DriverInstanceId` merged onto the cluster-level `DriverConfig`, so a literal `connectionString` pasted there leaks exactly as one pasted into the driver — the leak #498 closed for `DriverInstance.DriverConfig` and `Device.DeviceConfig`. Gated at the SAVE, not at the deploy — a deliberate departure from both options the issue offered: - `DraftSnapshot` carries no `ClusterNode` rows, and adding them would widen the snapshot and every builder of one for a single rule about a value that never enters the artifact. - More to the point, a deploy gate is the wrong instrument here. Node overrides are not in the artifact, so blocking a deploy would not stop the credential being stored — it is already in the database and replicated by then. Refusing the save is the only point where "refuse to store it" is literally true, which is #498's own framing: discarding a secret on read is not the same as refusing to store it. The check moves into a shared `SqlCredentialGuard` so the two enforcement points cannot drift; `DraftValidator` now calls it instead of its own private helper. Kept deliberately narrow — the `Sql` driver's `connectionString` only, and only for instance ids that ARE Sql drivers. The issue asked whether to generalise to credential-shaped keys across every driver type; not doing that, for the same reason #498 did not: a broader sweep would start refusing configs that are legitimate today for drivers which never made an indirect-credential guarantee, which is a regression rather than defence in depth. Widen per driver, as each gains its own contract. The error names the offending driver instance(s) and NEVER the value — it reaches the AdminUI and the audit trail. 14 new cases cover both halves, including the case variants (System.Text.Json binds `ConnectionString` to `connectionString`, so a case variant is the same key, not a bypass), non-Sql ids being ignored, and every blank/malformed/non-object shape. |
||
|
Joseph Doherty
|
549c656489 |
fix(alarms): re-assert non-normal scripted-alarm conditions after a (re)load (#487)
A deploy that triggers a FULL address-space rebuild clears `_alarmConditions` and re-materialises every condition node fresh — inactive, acked, confirmed, unshelved. The engine reloads its persisted state and re-derives Active from the predicate, so it ends up correct; but there is no transition to report, so `LoadAsync` yields `EmissionKind.None` and `OnEngineEmission` drops it. Nothing writes the node. Live-reproduced on the docker-dev rig against the pre-fix image: an alarm that had fired and cleared but was still UNACKNOWLEDGED came back from a rebuild reporting Acknowledged with Retain=false — so it disappeared from ConditionRefresh entirely, while the engine still held Unacknowledged. An operator's outstanding alarm silently vanishes from the alarm list on deploy. `ScriptedAlarmHostActor.ReassertConditionNodes` closes it. After each load it reads the new `ScriptedAlarmEngine.GetProjections()` and sends one `AlarmStateUpdate` per alarm NOT in the Part 9 no-event position. Load-bearing properties: - Node-only. It sends `AlarmStateUpdate` and nothing else — never the `alerts` topic, never the telemetry hub. `alerts` is the historization path, so a row per alarm per deploy would append a duplicate historian/AVEVA record every time anyone deploys. This is why it reads a projection rather than re-emitting: `ScriptedAlarmProjection` carries no `EmissionKind`, so there is nothing for the emission machinery — or a future refactor — to mistake for a transition. The issue's sketch said `GetState(alarmId)` + `LoadedAlarmIds`; that alone would have clobbered the node's severity and message, so the projection also carries the definition's severity, the resolved message template, and the last-observed worst-input quality. - Only non-normal alarms. One in the no-event position already matches what the materialise built, so a never-fired alarm stays byte-identical to a deploy without this pass (including its Message, which the materialise seeds with the display name), and an all-normal deploy writes nothing at all. - Timestamp is the condition's own `LastTransitionUtc`, not the deploy instant. - No duplicate Part 9 events: `WriteAlarmCondition`'s delta-gate compares against the node's CURRENT state, so a re-assert onto a freshly materialised node is a genuine delta (fires once — correct, the rebuild reset what clients see) while one onto a surgically-preserved node is suppressed. Tests: 6 new host tests + 6 new engine tests. Falsifiability checked by disabling the call — 4 of the 6 host tests go red; the other 2 are absence guards against an over-broad fix and pass either way by design. Live gate (docker-dev, central-2, pre-fix image vs. rebuilt image): - pre-fix: condition absent from ConditionRefresh after a rebuild; - post-fix: present, Unacknowledged, Retain=True, stamped with its own LastTransitionUtc rather than the restart instant; - /alerts stayed empty across two re-asserts, with the panel proven live by a real ACTIVATED transition immediately afterwards — no duplicate history. |
||
|
Joseph Doherty
|
1919a8e538 |
feat(config): reject a persisted Sql connectionString at the deploy gate (#498)
The Sql driver's "a pasted literal connection string is never persisted" guarantee was
enforced only on the READ path: SqlDriverConfigDto has no connectionString property and
UnmappedMemberHandling.Skip drops the key on deserialization. Nothing stopped it being
WRITTEN. Config blobs are schemaless JSON columns and the fallback authoring pattern for
a driver without a typed form is a raw-JSON textarea, so an operator pasting
{"connectionString":"Server=...;Password=..."} would put a live credential in the
ConfigDb — persisted, replicated to every node's artifact cache, readable by anyone with
config access — while the runtime silently ignored it and the driver failed to connect.
Discarding a secret on read is not the same as refusing to store it.
DraftValidator.ValidateSqlConnectionStringNotPersisted fails the deploy
(SqlConnectionStringPersisted) when a Sql-typed driver instance's DriverConfig, or the
DeviceConfig of any device beneath it, carries a top-level connectionString key.
- Both surfaces are checked because DeviceConfig is merged onto DriverConfig before the
DTO sees it — a credential pasted into the device blob is the identical leak.
- The key is matched case-insensitively: System.Text.Json binds ConnectionString to a
connectionString property by default, so an ordinal match would leave the obvious
bypass open.
- Only the top level is scanned; the DTO is flat, so a nested occurrence cannot bind and
is not the mistake this rule exists to catch.
- The message never echoes the value — validation errors reach the AdminUI, the deploy
log and the audit trail, and repeating the string there would leak the credential the
rule is refusing to store. Pinned by a test.
- Malformed/blank/non-object JSON never throws: a parse failure would take down every
other rule in the same pass.
16 tests. Verified falsifiable — with the rule disabled the 6 positive assertions fail
and the 10 negatives stay green, so none of them passes vacuously.
Design §8.2's "if an inline connectionString is ever permitted (dev convenience only),
the AdminUI redacts it" carve-out is withdrawn in the same change: redaction only hides a
credential that has already been written.
Not covered, and filed as a follow-up: ClusterNode.DriverConfigOverridesJson is a third
persistence surface for a driver config blob, but it is not part of DraftSnapshot, so
this validator cannot see it.
|
||
|
Joseph Doherty
|
8d9155682d |
fix(mqtt): browse-commit writes a bindable address + a Request-rebirth affordance
Two gaps the Task 23/24 review found, both blocking Task 26's live gate.
Gap 1 — browse-commit produced a silently dead MQTT tag. RawBrowseCommitMapper
had no `Mqtt` case, so a committed leaf fell through to the generic
`{"address": …}` key. Neither MqttTagDefinitionFactory entry point reads
`address`: the tag deployed clean and reported BadNodeIdUnknown forever, with
no signal at commit time. Predates Task 23 (Plain was affected too), but Task 23
built the Sparkplug metric tree precisely so an operator could browse and commit
a binding.
The address is a DESCRIPTOR, not a reference string, and it cannot be recovered
from the browse node id: `{group}/{node}[/{device}]::{metric}` where a metric
name legitimately contains `/` (`Node Control/Rebirth`) — the ambiguity
MetricSeparator's remarks already name. So the session STATES it, via a new
`AttributeInfo.AddressFields` seam, and the mapper reads it. Which keys are
emitted is also how the mapper learns Plain vs Sparkplug — the driver type
reaching it is just `Mqtt`, and the mode lives on the driver config. Key names
are single-sourced in the new `MqttTagConfigKeys` (producer, mapper, factory),
with the literals pinned by a test so a symmetric rename cannot silently unbind
already-persisted blobs. A leaf with no stated address is refused at commit in
words rather than committed dead.
Gap 2 — RequestRebirthAsync had no UI. Task 23 shipped it backend-only; Task 26
step 1 assumes the button, and Task 25's runbook notes the picker tree stays
empty until a birth lands, so it is the only way to fill it on demand. Added to
the /raw browse modal: scope = the clicked tree node (device/metric resolve up to
their edge node, group fans out and is refused whole past 32), an explicit
two-click confirm naming the resolved scope and its consequence, the outcome or
the refusal shown rather than swallowed. Offered only for a Sparkplug session
(new `IRebirthCapableBrowseSession.RebirthAvailable` — one session class serves
both modes, so a type test alone would draw a button that can only throw) and
only to a DriverOperator; the server-side gate remains the boundary.
Tests: MQTT 545 → 581, AdminUI 781 → 800. The round-trip tests feed the emitted
blob to the REAL factory and were falsified by mutating the emitted key name.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
|
||
|
Joseph Doherty
|
64ec7aa43a |
feat(mqtt): Sparkplug birth-driven browser tree + scoped Request-rebirth action
Unseals the Sparkplug branch of the MQTT address-picker browser that Task 10 deliberately sealed shut, and adds the first (and only) sanctioned publish on a browse session. - MqttBrowseSession serves Group -> EdgeNode -> [Device] -> Metric in Sparkplug mode, decoded from observed NBIRTH/DBIRTH certificates (the only Sparkplug messages that name their metrics). Node-level metrics hang directly under their edge node -- no synthesised device folder, because the authored address tuple genuinely has deviceId = null for them. Metric node ids use a '::' separator: a metric name may contain '/', so slash-joining would make a node-level metric indistinguishable from a device folder. - AttributesAsync is purely birth-derived in Sparkplug mode. The plain path's UTF-8 inference / payload-snippet machinery does not run and no payload bytes are retained: a Sparkplug body is protobuf and would be reported as "(N bytes, binary)" for every metric in the plant. A datatype ToDriverDataType cannot map is reported as the Sparkplug type name, never coerced. - RequestRebirthAsync(scope) is the one publishing member, reached only by an explicit operator action. It routes through the counted PublishAsync chokepoint via a private RebirthTransport adapter, so PublishCountForTest still proves that OpenAsync/RootAsync/ExpandAsync/AttributesAsync/Observe/ DisposeAsync publish nothing -- now in both modes. A device or metric scope resolves up to its owning edge node (NCMD is node-addressed); group scope enumerates observed edge nodes and is refused whole past MaxGroupRebirthNodes = 32. - BrowserSessionService.RequestRebirthAsync gates on the same DriverOperator policy that gates the picker, evaluated server-side where the action happens rather than only at render time, fails closed, and Info-logs the scope and the published count. Exposed through the new IRebirthCapableBrowseSession contract so "does this session write?" stays a type question. - ToBrowseOptions' Last-Will landmine re-verified: MqttDriverOptions still carries nothing will-shaped (an NDEATH is a broker-published will and would be invisible to PublishCountForTest), now pinned by a reflection guard. Falsifiability: injecting a publish into RootAsync reddens the Sparkplug and plain passivity tests; double-publishing per target reddens all four one-NCMD-per-node assertions. 572/572 MQTT tests, 24/24 AdminUI browsing tests. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
f79d13e2d8 |
feat(mqtt): factory + DriverTypeNames.Mqtt + host factory/probe registration
Task 9. Adds MqttDriverFactoryExtensions (DriverTypeName = DriverTypeNames.Mqtt, direct deserialization of MqttDriverOptions — no intermediate DTO, mirroring OpcUaClientDriverFactoryExtensions), the DriverTypeNames.Mqtt constant, and both host wiring sites: the factory in DriverFactoryBootstrap.Register and the probe in AddOtOpcUaDriverProbes (the admin-node path Program.cs calls in its hasAdmin block — a probe wired only on driver nodes makes Test-connect silently dead). Carried-forward items: - Converges every MQTT config-parsing seam onto ONE JsonSerializerOptions — MqttJson.Options, in .Contracts alongside MqttDriverOptions. It replaces the probe's internal JsonOpts and the browser's separate private copy; the factory, the probe, MqttDriver.ParseOptions and MqttDriverBrowser now all parse through it. .Contracts is the only assembly all four consumers reference, and the browser's reference to the runtime .Driver project is a documented layering exception scheduled for removal — anchoring the options there would resurrect the duplicate the day it goes away. - Replaces the "Mqtt" literals in MqttDriverProbe, MqttDriverBrowser and MqttDriver with the constant (string value unchanged). - Tightens MqttDriverProbeTests.ProbeAsync_EnumAsName: it asserted only Ok == false + non-empty message, which is also exactly what a JSON-parse failure produces — so it stayed green under the very regression it names. It now asserts the probe got past the parse and reached the network. Falsifiability: deleting JsonStringEnumConverter from MqttJson.Options reddens 9 tests across 4 suites, including the tightened probe test (message becomes "Config JSON is invalid: The JSON value could not be converted to MqttProtocolVersion") — which the pre-fix assertions would have passed. Also references the MQTT driver from Core.Abstractions.Tests so DriverTypeNamesGuardTests' reflective bin scan discovers the new factory and the constant/factory parity check stays honest. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
9b5a96876e |
feat(sql): register Sql factory + probe in Host, add DriverTypeNames.Sql
Add DriverTypeNames.Sql (+ All) as the shared source of truth, and wire the driver into the Host: register the factory in DriverFactoryBootstrap.Register via Driver.Sql.SqlDriverFactoryExtensions.Register(registry, loggerFactory), add the SqlProbe (= Driver.Sql.SqlDriverProbe) probe via TryAddEnumerable, and add the Driver.Sql project reference to the Host. Default tier A (SQL client is managed + cross-platform, so ShouldStub needs no change). Repoint the interim SqlDriver.DriverTypeName / SqlDriverFactoryExtensions.DriverTypeName literals at DriverTypeNames.Sql; both still resolve to "Sql", so the AdminUI TagConfigEditorMap/TagConfigValidator keys and the DriverTypeName-parity test stay green. The Core guard test discovers factories from its own bin, so it also gains a Driver.Sql project reference — that is the "registered-factory side" that lets DriverTypeNamesGuardTests' bidirectional-parity check pass (4/4 green). Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
279d1d0fb1 |
feat(cluster): simultaneous-cold-start split-brain guard (opt-in)
Closes the residual Phase-7 gap: when BOTH nodes of a 2-node pair cold-start at the same instant, each self-first seed runs FirstSeedNodeProcess, times out waiting for the other, and forms its own 1-node cluster — two Primaries in one pair (the Phase 6 live gate reproduced this reliably on docker). The prior mitigation was operational (staggered start / compose depends_on), which does not exist on production hardware. The guard (dark switch Cluster:BootstrapGuard:Enabled, default OFF) prevents the split without giving up cold-start-alone: - The lower-address node is the preferred founder: self-first, forms immediately. - The higher node probes its partner's Akka port (TCP connect) up to PartnerProbeSeconds: reachable => peer-first (join the founder, never race it); unreachable => self-first (partner is dead, form alone). The order is decided BEFORE the single JoinSeedNodes, from an explicit reachability signal — never re-formed mid-handshake (the retired SelfFormAfter failure mode). When on, Akka gets no config seeds (BuildClusterOptions) and ClusterBootstrapCoordinator drives the join. Review-driven hardening: case-insensitive tie-break (a hostname-casing mismatch would reopen the split); fail-fast validation of the timing knobs; the residual "founder dies in the probe->join window" hang is documented and made operator-visible (warning + a restart recovers). Real-ActorSystem coordinator tests cover the load-bearing higher-node cold-start-alone case; 147/147 Cluster tests pass. Live-gated on docker-dev (site-a = enablement demo, serialization removed, guard on; site-b keeps depends_on-serialization + guard off as the A/B control): simultaneous start -> site-a-1 founds, site-a-2 probes-reachable-joins -> 250/240, NO split; higher-node-alone -> probes dead founder, self-forms -> 250; founder rejoin -> 240. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
344b0d3334 |
fix(cluster): validate split topology against EFFECTIVE roles, not raw Cluster:Roles
Code review found SplitTopologyTransportValidator read the wrong role view,
defeating its fail-loud purpose two ways:
(a) Roles-source divergence. AkkaClusterOptions.Roles binds Cluster:Roles
but falls back to OTOPCUA_ROLES (RoleParser.Parse) when it is empty. A
node configured with only OTOPCUA_ROLES=driver,cluster-SITE-A genuinely
carries the cluster role in Akka, yet the validator saw no cluster role
and passed silently on MeshTransport:Mode=Dps (the documented incident).
(b) Case. The Cluster:Roles bind path is verbatim while RoleParser.Parse
lowercases the OTOPCUA_ROLES path; 'Cluster-SITE-A' / 'Driver' / 'Admin'
slipped past the ordinal checks.
Resolve effective roles the same way the node does — Cluster:Roles, else
RoleParser.Parse(env OTOPCUA_ROLES) exactly as Program.cs — then normalize
both (trim + ToLowerInvariant) before IsClusterRole/driver/admin. The message
still names the ClusterId in the operator's original spelling. Exemption and
all other behaviour unchanged. Contained to the validator.
Tests: OTOPCUA_ROLES-only cluster role on Dps fails; mixed-case roles on Dps
fail; cluster role with neither admin nor driver needs only ClusterClient;
Cluster:Roles wins over a stray OTOPCUA_ROLES. 16/16 green (124/124 project).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
|
||
|
Joseph Doherty
|
a886d5e6e0 |
feat(cluster): refuse to boot a split-topology node on a DPS transport mode
Per-cluster mesh Phase 6 (Task 5). A node carrying a cluster-{ClusterId}
role (RoleParser.IsClusterRole) is in the split topology, where the
Phase 2/3/5 DPS dark-switch branches deliver nothing across the mesh
boundary. SplitTopologyTransportValidator fails host start unless such a
node uses the mesh-crossing transports: MeshTransport:Mode=ClusterClient
always; Telemetry:Mode=Grpc if it has the driver role; TelemetryDial:Mode
=Grpc if it has the admin role. It is a no-op for any node with no
cluster-role (legacy / single-mesh / test). Roles + the three modes are
cross-read from IConfiguration, mirroring ConfigSourceOptionsValidator.
Registered via AddValidatedOptions on MeshTransportOptions with
ValidateOnStart, alongside the sibling validators.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
|
||
|
Joseph Doherty
|
a4d3ab4005 |
docs(mesh): fix stale redundancy-singleton comments + prove the driver fallback end-to-end
Review follow-ups on the redundancy re-home (Phase 6, Task 2):
- Program.cs: the hasDriver comment claimed IClusterRoleInfo "throws at host
start if this driver node carries no cluster role" — false since the fallback
landed. Rewrite it to describe the cluster-{ClusterId} scope with the driver-role
fallback for legacy single-mesh / not-yet-migrated nodes.
- ClusterRoleInfo.cs: the SubscriberActor comment described RedundancyStateActor as
the "admin-role singleton" — stale. Note it is now a cluster-{ClusterId}-scoped
singleton spawned on every driver node (LeaderChanged stays a no-op here).
- Add a host/registry-level test for the driver-role FALLBACK path (roles ["driver"],
no cluster role) closing the coverage asymmetry — it was proven only in the pure
helper. Asserts the node boots and registers RedundancyStateActorKey, i.e. the boot
the earlier throw-based version would have aborted.
The boot helper now supplies an in-memory IDbContextFactory and a fake IClusterRoleInfo:
Akka.Hosting invokes singleton props factories eagerly at StartAsync, and the admin
ClusterNodeAddressReconciler factory reads both (the latter for its per-cluster reconcile
scope, added by a sibling Phase 6 task) — without them the admin boot NREs.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
|
||
|
Joseph Doherty
|
2d2f1decae |
feat(mesh): ClusterRoleInfo exposes the node's own cluster-scoped role
Phase 6 Task 1. IClusterRoleInfo gains ClusterRole/ClusterId, derived from the node's configured AkkaClusterOptions.Roles at construction time (not live Cluster.State) so the identity is available before the cluster forms. First cluster-scoped role wins when more than one is configured, logged as a Warning. Updates the FakeClusterRoleInfo test double in ServiceCollectionExtensionsTests to satisfy the new interface members. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
87ff00fd30 |
feat(mesh): RoleParser accepts cluster-{ClusterId} roles; consolidate the driver-role literal
Phase 6 (per-cluster mesh split) needs nodes to carry a cluster-scoped role like cluster-SITE-A; RoleParser previously rejected anything outside the fixed admin/driver/dev set. Adds IsClusterRole/ ClusterIdFromRole plus well-known-role constants, and points the three duplicate "driver" string literals (RedundancyStateActor, ClusterNodeAddressReconcilerActor, ServiceCollectionExtensions) at the new RoleParser.Driver so the value has one source, without renaming any existing DriverRole symbol. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
53ae0100a2 |
feat(mesh-phase5): Telemetry/TelemetryDial options + fail-closed validators
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
a845a6d2fd |
feat(mesh-phase5): telemetry.proto contract + oneof + contract-lock test
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
3a590a0cb7 |
refactor(mesh-phase4): retire EfAlarmConditionStateStore + drop dead ScriptedAlarmState table (Task 9)
Scripted-alarm condition state lives in LocalDb (Phase 4); the ConfigDb-backed Ef store + ScriptedAlarmState table are now dead. Removes the test-harness Ef fallback (a DB-backed node with no store now skips the alarm host), deletes the store + entity + model config, and drops the table via migration. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
4f4cdd05ec |
feat(mesh-phase4): LocalDb alarm-condition-state store (replicated, pair-local)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
d630a7e267 |
feat(mesh-phase4): driver-only node must be FetchAndCache (validator)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
62e8f54f71 |
feat(mesh): deployment_artifact.v1 proto + first in-repo Grpc.Tools codegen
The repo has consumed packaged gRPC clients (LocalDb.Replication, HistorianGateway.Client) but never compiled a proto locally. This adds the Grpc.Tools toolchain to Commons and a deployment_artifact.proto with a single server-streaming Fetch(deployment_id) -> stream ArtifactChunk RPC, generating both the server base and the client stub (GrpcServices="Both") so central (Host/AdminUI) and the node (Runtime) share one reference. A compile-touch test pins the codegen: it stops compiling if generation regresses. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
9d1e60c00c |
feat(mesh): ConfigSource/ConfigServe options + validator (Phase 3 dark switch)
ConfigSource:Mode = Direct (default, read central SQL) | FetchAndCache (fetch the artifact from central over gRPC, read LocalDb). The validator fails host start on a FetchAndCache shape that cannot fetch — no endpoints, a non-http(s) endpoint, no shared key, or a non-positive timeout — because each otherwise surfaces as a silent absence (a deploy that never applies). ConfigServe (the central serve surface) takes a plain Configure: a 0 port is a valid "disabled". Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
d5b5cb6ede |
feat(mesh): CentralCommunicationActor — dark-switched fan-out, DB-sourced contacts
Phase 2 Task 3. Central-side end of the boundary: routes every central->node command out over DPS or ClusterClient per MeshTransport:Mode, and forwards inbound ApplyAcks to the deploy coordinator singleton (Forward, not Tell, so the coordinator sees the node rather than this relay). Three things here are load-bearing and easy to get wrong later: SendToAll, never Send. Today's DPS publish reaches EVERY DriverHostActor and the node side has no ClusterId or node filter to compensate -- scoping happens later, inside the artifact. ClusterClient.Send delivers to exactly ONE registered actor, so it would deploy to a single node while every other node silently kept its old config, and the deployment could still seal green. Sabotage-verified: swapping to Send reddens exactly that one test. Exactly ONE ClusterClient, fleet-wide. A receptionist serves its whole cluster, so SendToAll reaches every registered node-comm actor in the mesh regardless of which contact point was dialled. One client per application Cluster -- the shape Phase 6 wants -- would fan each command out once per cluster while the fleet is still one mesh: N x duplicate DispatchDeployment and ApplyAck. Marked TODO(Phase 6). The contact set does NOT scope delivery. Excluding a maintenance-mode node from the contacts does not stop it receiving commands; it still receives them, as it does under today's broadcast. The filter is about which receptionists are worth dialling, not about who gets the message. Stated in the code because the opposite is the natural assumption. Also carries the sister project's shipped fixes: per-row address parsing so one malformed ClusterNode cannot abort the refresh and leave the contact set half-built (regression test orders the bad row FIRST), the cache entry cleared before recreate so a failed create cannot leave commands routing into a stopping actor, and a Status.Failure handler so a DB outage is a Warning rather than a silent debug-level unhandled message. Two test-harness bugs found and fixed while writing this, both mine: SubscribeAck goes to the SENDER (so the mediator subscribe needed an explicit sender), and EventFilter matches case-INSENSITIVELY, so a "was NOT" filter also caught this actor's own "was not created" warning. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
0d6669d5ff |
feat(mesh): MeshCommand envelope + generation-suffixed ClusterClient factory
Phase 2 Task 2. Two small pieces the comm actors are built on. MeshCommand carries a command plus the DPS topic it would have been published on, so the admin singletons stop knowing which transport is in force. Without it the dark switch would have to be threaded through five publish sites across two actors, each free to drift. Central-side only -- it never crosses the wire, so it carries no serialization contract. DefaultMeshClusterClientFactory appends a generation counter to the actor name. Context.Stop is asynchronous and the name stays reserved until termination completes, so recreating the same name inside one message handler throws InvalidActorNameException -- which is exactly what a contact-set change does (stop the old client, create the replacement, one handler). Ported from the sister project, where it was a shipped bug fix rather than foresight. Sabotage-verified: freezing the name to a constant reddens both name tests with InvalidActorNameException. Dropped a third test I had written -- it asserted ClusterClientSettings directly and never touched the factory, so it was coverage theatre. Replaced with a comment saying where contact propagation IS covered (the Task 8 boundary test, the only place it can actually fail). Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
7b71a6a35d |
feat(mesh): receptionist extension, zero ClusterClient buffering, frame-size logging
Phase 2 Task 1. Three changes to the embedded base config, plus a test that reads every one of them back off a RUNNING ActorSystem rather than off the HOCON text -- the settled idiom here (SplitBrainResolverActivationTests), because several fragments compete and the losing one still reads correctly in the file. - Registers the ClusterClientReceptionist extension. It is not auto-started, so without this the boundary only listens if some code path happens to call ClusterClientReceptionist.Get(system) first: "is the boundary up?" would depend on startup ordering rather than on configuration. - buffer-size = 0. This is the one genuine behaviour change and it is NOT what the sister project runs -- they never wrote this section and inherit 1000. Buffering would replay a DispatchDeployment on reconnect and apply a deployment the coordinator already sealed as TimedOut, leaving the node on a configuration the database says failed. - log-frame-size-exceeding = 32000b. An oversized frame is dropped WITHOUT tearing down the association: heartbeats keep flowing, the node reports healthy, and the only symptom is a command that never arrived. Our deploy notify is payload-free, so this canary should stay quiet. reconnect-timeout and receptionist.role restate Akka defaults; they are stated explicitly and pinned because the behaviour is load-bearing, and the comments say which is which. One test was VACUOUS on first write: Config.GetInt returns 0 for a missing key, so the buffering assertion passed before the feature existed -- it could not distinguish "explicitly 0" from "absent, default 1000 in force". Rewritten to assert through ClusterClientSettings/ClusterReceptionistSettings, the objects the client is actually built from. Sabotage-verified after the fix: setting buffer-size back to 1000 reddens exactly that one test. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
5439f14804 |
feat(mesh): MeshTransportOptions — dark switch between DPS and ClusterClient
Phase 2 Task 0. Binds a validated `MeshTransport` section selecting which transport carries central->node commands, defaulting to Dps so nothing changes until the rig gate passes. The validator exists because every fault it catches otherwise surfaces as an ABSENCE -- a deployment that never arrives, an alarm ack that does nothing -- which has no stack trace and no failing node to point at. Two of the shapes it rejects are ported from the sister project's shipped mistakes: a contact point carrying an actor-path suffix, and (documented in the options XML) a template listing the node's OWN remoting port as a central contact, which is a permanent failure in the initial-contact rotation. Contact points are required only under ClusterClient mode; requiring them under the default would make the section mandatory on admin-only nodes that have no reason to carry them. Sabotage-verified: relaxing the empty-contacts guard and the actor-path-suffix guard turns exactly those two tests red, and nothing else. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
ee69caa270 |
feat(config): add ClusterNode.MaintenanceMode — the hatch the live gate proved missing
Phase 1 gate step 4 failed: setting Enabled = 0 to take a node out of service returns 422 ClusterEnabledNodeCountMismatch, because DraftValidator.ValidateClusterTopology requires the enabled-node count to equal ServerCluster.NodeCount. On a Warm/Hot pair — every cluster on the rig, and every cluster in the target topology — Enabled can therefore never be the maintenance hatch. The plan called step 4 "the check that makes step 3 acceptable to ship", so Task 3's behaviour change was not shippable as it stood: a node down for maintenance would block every deployment to its cluster. The two rules were each reasonable and contradictory together — the validator reads Enabled as "part of the declared topology", Phase 1 additionally read it as "expect an ack". Split the meanings rather than weaken either rule: Enabled part of the declared topology (validator, untouched) MaintenanceMode expected to participate now (coordinator + reconciler) Rejected alternatives: counting configured rather than enabled nodes (drops the guard against booting a pair into InvalidTopology); downgrading the rule to a warning (weakens a deploy gate for everyone); shipping with no hatch. Sabotage: dropping !n.MaintenanceMode from the coordinator query turns the new test red. It also asserts both nodes remain Enabled, so the fix cannot quietly regress to disabling the row after all. Configuration.Tests 95/95, ControlPlane.Tests 101/101, solution builds clean. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
da74ebd696 |
feat(config): migration AddClusterNodeTransportPorts
Additive only — two AddColumn ops, no AlterColumn, so no accumulated model drift is riding along with this change: ALTER TABLE [ClusterNode] ADD [AkkaPort] int NOT NULL DEFAULT 4053; ALTER TABLE [ClusterNode] ADD [GrpcPort] int NULL; Adds a third test covering the DB-side default specifically. The entity round-trip cannot see it: ClusterNode.AkkaPort's CLR initializer is also 4053, so that assertion passes with the mapping default deleted. The new test inserts through raw SQL with the column omitted, so only the schema can supply the value — verified by setting defaultValue: 0, which turns exactly that one test red and leaves the other two green. Configuration.Tests 95/95. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
2bbb02713c |
feat(config): add ClusterNode.AkkaPort/GrpcPort — central's dial targets
Per-cluster mesh Phase 1 groundwork. Once the meshes split (Phase 2) central can no longer see a site node's appsettings, so the transport ports it must dial have to live in a row central can read. AkkaPort is non-nullable with a 4053 default — every node listens on a remoting port, so 0 is never a truthful value and pre-existing rows must migrate to something real. GrpcPort is nullable with no default: nothing listens on it until Phase 5, and a non-null default would assert a port that does not exist. Both are documented as central's dial targets rather than the node's own binding config; the duplication against Cluster:Port is reconciled in Task 4. The new SchemaCompliance test is red until the migration lands (Task 2). Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
3f24d4d6bf |
feat(cluster)!: self-first seed ordering replaces the InitJoin self-form watchdog
Akka runs FirstSeedNodeProcess -- the only bootstrap path that can form a NEW
cluster when no peer answers InitJoin -- exclusively when seed-nodes[0] is the
node's own address. Every other node runs JoinSeedNodeProcess and retries
InitJoin forever. That is why "both peers are seeds" never meant "either can
cold-start alone", and it is fixed here the way Akka itself intends: each seed
node lists ITSELF first.
- docker-dev central-2 now lists itself as SeedNodes__0 (central-1 was already
self-first). The site nodes are untouched -- they seed off central-1 only and
are deliberately not seeds.
- AkkaClusterOptionsValidator enforces the invariant at boot via the shared
ZB.MOM.WW.Configuration AddValidatedOptions/ValidateOnStart seam. The rule is
CONDITIONAL -- self must be seed[0] only IF self is in the list at all -- or it
would refuse to boot every driver-only site node. Identity is compared on
PublicHostname (falling back to Hostname when blank) AND port, i.e. the address
Akka puts in SelfAddress: matching the 0.0.0.0 bind address would find no seed
anywhere and leave the rule silently inert on the whole docker-dev rig.
- ClusterBootstrapFallback + Cluster:SelfFormAfter are DELETED. The watchdog sat
outside Akka's join handshake, so it could not distinguish "no seed answered"
from "a seed answered and the join is in flight" -- and Cluster.Join(SelfAddress)
is not ignored mid-handshake, it wins. The live gate (
|
||
|
Joseph Doherty
|
ea45ace1c3 |
fix(cluster): don't self-form while a seed peer is reachable — the fallback islanded a restarting node
The live gate for the manual-failover control caught the self-form fallback forming a SECOND cluster. A node bounced by a failover restarted, received InitJoinAck from its live peer — the join was in flight and healthy — but did not get the Welcome inside the 10s window, because the peer's ring still held the node's previous incarnation (Exiting -> Down -> Removed). The fallback fired on the timer and the node islanded itself until an operator restarted it. The original design assumed Cluster.Join(SelfAddress) would be ignored mid-handshake. It is not — it wins. And since manual failover deliberately produces that restart, every failover could island the node it bounced. Before self-forming, TCP-probe the other seed addresses; a reachable peer means wait another window instead. That is also what the fallback claims to detect: 'no seed answered InitJoin (peer down at boot)'. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
9ded86342e |
feat(cluster): InitJoin self-form fallback armed via Akka.Hosting startup task
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
979dc102dd |
feat(cluster): SelfFormAfter option (bootstrap self-form fallback window)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
0de1352e55 |
fix(deps): pin System.Security.Cryptography.Xml 10.0.10 — fresh restores were red
Four NU1903 high-severity advisories (GHSA-23rf-6693-g89p, GHSA-8q5v-6pqq-x66h, GHSA-cvvh-rhrc-wg4q, GHSA-g8r8-53c2-pm3f) landed in the NuGet audit data against System.Security.Cryptography.Xml 10.0.7, which Microsoft.AspNetCore.DataProtection 10.0.7 pulls in transitively for key storage. Under TreatWarningsAsErrors a fresh restore produces 204 NU1903 errors and the solution does not build at all. This was invisible locally: machines with cached audit data keep building, so the break only shows on a clean clone or a docker image build. It surfaced here by accident — a detached worktree at the pre-change baseline, created to check whether a flaky test predated today's work, could not restore. Worth noting that the diagnostic path was the accident, not the intent: nothing in the normal build or test loop would have reported this before someone else hit it. Same surgical-pin pattern as the SQLitePCLRaw entry directly above it: a direct PackageReference at the one project where the chain enters the repo (Core.Configuration), rather than CentralPackageTransitivePinningEnabled, which this repo already documents as breaking the Roslyn version split. Bumping the DataProtection parent to 10.0.10 instead was rejected for the reason the sister repo recorded when it hit the same advisories: 10.0.10 floors Microsoft.Extensions.* and, via the EFCore adapter, Microsoft.EntityFrameworkCore at 10.0.10, forcing a family-wide servicing bump and an NU1605 downgrade cascade that deserves its own reviewed change. Verified the way the defect demanded — in a fresh worktree, not this one: restore clean (0 errors, down from 204) and full solution build clean. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
50b55ee4b9 |
fix(redundancy): elect the Primary by cluster age, not by address
RedundancyStateActor derived the driver Primary from ClusterState.RoleLeader("driver").
Akka offers two different notions of a "first" member and they are not the same
one: role leader is the lowest-ADDRESSED Up member (host, then port), while
ClusterSingletonManager places singletons on the OLDEST — lowest up-number.
They agree on a freshly-formed cluster, which is why every existing test passed.
They diverge after any restart: the restarted node re-joins as the youngest while
keeping its address, so if it holds the lower address it becomes role leader while
the singletons stay put. The snapshot would then name a Primary that is not
hosting the work, and every Primary-gated surface follows it — inbound device
writes, native-alarm acks, the fleet-wide alerts emit, and the alarm-history
drain would all enable on the wrong node while the node actually running the
singletons stayed gated off.
BuildSnapshot now selects the oldest Up member carrying the driver role, matching
singleton placement. Leaving members are excluded: a node handing its singletons
over must not be named Primary.
NodeRedundancyState.IsRoleLeaderForDriver is renamed IsDriverPrimary, and
NodeHealthInputs.IsDriverRoleLeader likewise. Keeping the old names would have
left the wire contract asserting a derivation the code no longer uses — the same
drift that made this defect invisible.
Proven by a real two-node cluster rather than a mock. RedundancyPrimaryElectionTests
binds the first-joining node to the HIGHER port, so oldest and lowest-address name
different nodes, and includes a fixture assertion that the divergence actually
occurred — without it the real assertion could pass for the wrong reason. Positive
control: restoring the RoleLeader derivation turns exactly the two election tests
red while the fixture check stays green.
Runtime.Tests 440 passed, ControlPlane.Tests 82, Cluster.Tests 36.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
|
||
|
Joseph Doherty
|
2964361a6f |
fix(cluster): auto-down downing strategy — a two-node pair could not survive an oldest-node crash
Ports the sister project's live-proven fix (ScadaBridge cf3bd52f). OtOpcUa ran the identical configuration it indicts: keep-oldest with down-if-alone = on. Akka.NET 1.5.62's KeepOldest.OldestDecision only lets the down-if-alone branch rescue a side holding >= 2 members. A two-node cluster losing a peer is always a 1-vs-1 split, so the branch never fires, the lone survivor falls through to DownReachable and downs ITSELF, and run-coordinated-shutdown-when-down then terminates it. down-if-alone is a 3+-node feature and does not do what its name suggests for a pair: a crash of the oldest node is a total outage, which is the exact failure the redundancy pair exists to absorb. Cluster:SplitBrainResolverStrategy now selects the provider, defaulting to auto-down: Akka's AutoDowning with auto-down-unreachable-after = 15s, so the leader among the reachable members downs the unreachable peer and a crash of either node fails over in place. keep-oldest remains available for deployments that would rather take an outage than ever run dual-active during a real partition. An unrecognised value fails the host at startup rather than falling through to the fatal default. The tests assert the EFFECTIVE configuration — they start a real host through WithOtOpcUaClusterBootstrap and read akka.cluster.downing-provider-class back off the running ActorSystem. This is not incidental. The first draft inlined the three calls the bootstrap makes instead of calling it, which pinned the test's own wiring: sabotaging production's HOCON precedence left all 36 green. The prior file had the same shape at a smaller scale, asserting only that BuildClusterOptions returned a KeepOldestOption — true, and true of a configuration that cannot fail over. Positive control: making BuildDowningHocon emit nothing for auto-down turns exactly the two effective-config guards red. Measured while verifying rather than assumed: HoconAddMode.Append also wins here, purely because it is added last, so the mode name is not the guarantee. The comment now says so instead of asserting a precedence rule that does not hold. Not yet live-drilled on OtOpcUa. The docker-dev rig is a single six-node mesh where the 1-vs-1 pathology cannot occur; the kill-the-oldest drill belongs with the per-cluster mesh work that makes every mesh exactly two nodes (design doc 6.2 / Phase 0a). Recorded as an outstanding gate in docs/Redundancy.md. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
af545efdf5 |
feat(localdb): one-time alarm-historian.db migrator
Copies the pre-consolidation store-and-forward queue into the consolidated
database on first boot, then renames the legacy file aside. Rows are in that
file precisely because the historian could not be reached, so dropping them
on upgrade would discard exactly the alarm audit trail the queue exists to
protect.
Runs last in OnReady, after every RegisterReplicated call. It is the only
thing in OnReady that writes rows, and capture is trigger-based: a migration
that ran before registration would recover the backlog locally and never
replicate a line of it, silently and permanently.
Ids are derived from the payload rather than the plan's mig-{node}-{legacyId}
scheme. Node-prefixing solves the collision the legacy AUTOINCREMENT key
would cause -- node A's row 7 and node B's row 7 are different alarms -- but
it preserves a duplication that should be collapsed instead. A warm pair's
two legacy files OVERLAP: HistorianAdapterActor default-writes while its
redundancy role is unknown, so both nodes accepted the same transitions
during every boot window. Prefixed ids would carry those duplicates into the
merged buffer forever; equal-payload ids converge them. The same property
makes a crash between commit and rename harmless under INSERT OR IGNORE.
OnReady now takes IConfiguration rather than offering an overload that skips
the migration. A wiring mistake that silently discarded a node's undelivered
alarm history is not a mistake worth making possible.
The copy is restricted to the columns the legacy table actually has. Naming
a column an older build never wrote throws "no such column", which would
discard every row in the table rather than the one field.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
|
||
|
Joseph Doherty
|
71379816e7 |
feat(localdb): alarm S&F on LocalDb with primary-gated drain (cutover)
Moves the alarm store-and-forward buffer out of its own alarm-historian.db and into the node's consolidated LocalDb, where it replicates to the redundant pair peer. A node that dies holding undelivered alarm history no longer takes it to the grave. Tasks 2 and 3 land together, as the plan anticipated. They are not separable: the gate is a constructor argument of the rewritten sink, and a commit that replicated the queue without gating the drain would be a commit in which both nodes of a pair deliver every alarm event, continuously. That drain gate is the load-bearing part of the change, and the recon explains why it is new work rather than a refinement. Exactly-once delivery across a pair is enforced today on the ENQUEUE side, by HistorianAdapterActor: only the Primary enqueues, so the Secondary's queue is empty and its ungated drain has nothing to send. Replicating the table destroys that invariant. The gate is a Func<bool> the caller supplies, because the drain runs on a timer the sink owns rather than on a mailbox, and Core.AlarmHistorian cannot reference PrimaryGatePolicy in Runtime. Runtime supplies it via a new IRedundancyRoleView singleton that DriverHostActor publishes its Primary-gate verdict to -- the same verdict the inbound-write and native-ack gates use, so there is no second notion of am-I-the-Primary to drift. Two failure modes are deliberately closed: - The view is seeded OPEN, matching the policy's own answer for an unknown role with no driver peer. A deployment that runs no redundancy never publishes to it, and defaulting closed would silently stop its alarm history forever. - A gate that throws is read as not-now, never as permission, and a closed gate reports the new HistorianDrainState.NotPrimary rather than Idle. A Secondary's rising queue is supposed to look different from a stalled drain, and if BOTH nodes report NotPrimary the pair is misconfigured and says so instead of quietly filling toward the capacity ceiling. Row ids are a hash of the payload rather than fresh GUIDs. Both adapters accept the same fanned transition in the window before the first redundancy snapshot arrives, and under last-writer-wins an equal key converges those two accepts into one row instead of duplicating them. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
8a9cb40a72 |
feat(localdb): alarm_sf_events replicated table
Adds the alarm store-and-forward buffer to the consolidated LocalDb file and registers it for replication, so a node that dies with undelivered alarm history no longer takes it to the grave. The table keeps the legacy queue's shape rather than the status column the plan sketched. An acknowledged row is deleted, not marked delivered: that needs no second sweeper to keep the table bounded, leaves the capacity semantics untouched, and the replication engine carries the delete as a tombstone so the peer drops its copy anyway -- which is what the status column was for. last_error is retained because it is the only operator-facing record of why a row was dead-lettered. The primary key is app-minted TEXT. The legacy AUTOINCREMENT RowId cannot replicate under last-writer-wins: two nodes would independently allocate rowid 7 to different alarms and silently overwrite each other. The drain index therefore orders by enqueued_at_utc rather than insertion order, with id as a tiebreak so the ordering is total. Tables are created unconditionally, independent of AlarmHistorian:Enabled. An empty registered table costs three triggers; creating it lazily would mean a node that enables the historian later writes rows before its capture triggers exist, which is exactly the silent-loss shape OnReady's ordering comment warns about. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
a27eff3298 |
feat(localdb): boot from the pair-local artifact cache when central SQL is unreachable
Hooks the Bootstrap catch that previously went straight to Stale. On a cache hit the node serves its last-known-good configuration through the outage instead of coming up with an empty address space; on a miss, behaviour is byte-for-byte what it was. The read is unkeyed because ClusterId is only derivable from an artifact you already hold or from the unreachable central DB (recon D-1). Newest pointer wins, which is correct in every real topology - a node belongs to one cluster and its peer replicates that same row. Splits PushDesiredSubscriptionsFromArtifact out of PushDesiredSubscriptions: the cache path runs precisely when the ConfigDb read cannot succeed, so re-reading the artifact there would fail by definition. RunningFromCache is surfaced on NodeDiagnosticsSnapshot (optional param, existing call sites unaffected). A node running from cache looks entirely healthy from outside - full address space, live values - but its config is frozen and no deploy can reach it. It clears only on a real apply from central. |