#520 stage 6 (master 2026-08-01, live-gated). With Security:Acl:Enabled ON,
every data-plane operation needs a matching NodeAcl grant, not just writes /
alarm acks / HistoryRead. The stage-5 entry told ScadaBridge that a read-only
consumer was 'unaffected either way' — that now holds only while the flag is
off, which is the default.
Also records the two stage-6 behaviours a consumer would otherwise discover the
hard way: a deep Browse grant makes its ancestor folders browsable (so it
exposes their names), and revoking a grant stops a live subscription — one
BadUserAccessDenied then silence, resuming by itself if the grant returns, with
no reconnect needed.
Default is off, so a read-only address-space consumer is unaffected. With it on,
writes / alarm acks / HistoryRead need a matching grant for the principal the
session authenticates as — anonymous unless ScadaBridge is issued a key.
Reads, browses and subscriptions stay ungated in both modes.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
Gitea #520 (OtOpcUa master, merge fbe7403c) replaced LdapOpcUaUserAuthenticator
with an API-key authenticator on the OPC UA data plane. The index still told a
consumer that enforcement was "LDAP roles plus a write gate", which is now the
wrong instruction for anyone wiring a client.
Records what a consumer actually needs: the UserName/`apikey`/`otua_...` token
shape, that LDAP survives for AdminUI login only, that anonymous still defaults
to admitted (so ScadaBridge is not broken today), and that revocation is
asymmetric because keys reach driver-only nodes inside the deployment artifact.
The "ACLs authored but not enforced" warning stands -- stages 3, 5 and 6 remove
it.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
All three live-gated and merged to OtOpcUa master. The index still said write-through was
deferred and every MQTT node read-only, which #508 changed on 2026-07-31; #511 and #515
landed in merge 857a10b4. Two of the three are consumer-visible to ScadaBridge: MQTT nodes
now materialize Operate rather than ViewOnly where a write destination is declared, and a
Sparkplug UDT surfaces as ordinary scalar nodes instead of being skipped.
Both were recorded as open keep-or-delete decisions; both closed on OtOpcUa master abda6f9e.
Neither changes the OPC UA wire contract, so neither is consumer-facing to ScadaBridge — but
the index asserted them as current state, so it would have drifted.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
The entry claimed the periodic tag-set drift check had shipped. It was taken back
out two days later and is not in master. Corrected in place rather than deleted,
so the record shows both what landed and what became of it.
Carries forward the consequence a reader needs: AbCip and FOCAS browse a live
backend but implement no IRediscoverable, so they now have no tag-change signal
at all. Galaxy/TwinCAT/MTConnect/Mqtt raise natively and are unaffected. Still
not consumer-facing to ScadaBridge -- no OPC UA wire-contract change either way,
and the IRediscoverable operator prompt is untouched.
Also records the rest of that merge: 13 stale skipped tests retired, a reflection
guard over the deploy-time TagConfig gate, three silently-failing OpcUaClient
integration tests fixed, and the two missing driver guides written.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
The index recorded this as unmerged on a feature branch; it landed on master
2026-07-28. Also records the two follow-ups the first pass had deliberately
left unbuilt and which have since shipped: Sql/FOCAS in-place config re-parse
(made atomic, so a config change cannot half-apply a new tag set against an
old connection or backend), and tag-set drift detection for the four
genuinely-browsable drivers.
Neither is consumer-facing to ScadaBridge — no OPC UA wire-contract change, and
the drift signal is advisory rather than an address-space rebuild. The
previously-noted operationally-visible item stands: a DriverConfig edit now
costs a driver reconnect.
Corrects a fact this index recorded two days ago as consumer-relevant to
ScadaBridge. "IRediscoverable is inert fleet-wide" no longer holds: it is
consumed as an operator re-browse prompt on /hosts, carried on
DriverHealthChanged. The consumer-facing conclusion is unchanged and worth
keeping explicit — the served address space still only changes on a config
redeploy, because the prompt is advisory by design.
Gitea #507 (discovered-node injection) is closed as superseded and its code
deleted; it read two structures that are structurally empty in v3, so it could
never have injected anything. #521 tracks the still-unconsumed
IHostConnectivityProbe.
Adds the remediation status for the deferment register (branch
feat/deferment-remediation, tip 88ce8df0, unmerged). One item there is
operationally visible beyond OtOpcUa: #516 makes a changed DriverConfig
stop+respawn the driver rather than an in-place delta, so a config edit now
costs a driver reconnect — previously five drivers ignored the edit entirely
while the deployment still sealed green.
Corrects four stale claims about OtOpcUa and adds two consumer-relevant facts,
all source-verified against the OtOpcUa repo (not its docs):
* "Phases 0a..6 done, 7 not started" -> PROGRAM COMPLETE. Phase 7 merged
c50ebcf7 (6/6 failover drills) + bootstrap guard d1dac87f, both on master
and pushed 2026-07-24.
* The auto-down 1-vs-1 crash-the-oldest live gate is no longer "still open" /
"deferred to Phase 7" — Phase 7 drill D4 closed it; every survivor stayed
Up without self-downing.
* Phase 6's live gate was NOT "deferred by owner choice" — it ran and caught
three fix-forward defects, including a boot-crash StackOverflow on every
node (IClusterRoleInfo resolved inside the AddAkka configurator while the
ActorSystem was still building).
* Records the MAIN-only manual-failover constraint and the default-OFF
bootstrap guard's founder-election shape.
Adds OtOpcUa/deferment.md (e08855fb) and its two ScadaBridge-relevant entries:
* Node ACLs are authored, shipped in every artifact, and NOT ENFORCED — the
evaluator subsystem has zero production consumers and OtOpcUaNodeManager
never references it. Reads are ungated; authored deny rules have no effect.
Do not assume OtOpcUa filters an address space per-user.
* IRediscoverable is inert fleet-wide (#518/#507) — nine drivers raise it,
nothing consumes it, so a Galaxy redeploy / PLC re-download / Agent restart
does not change the served address space without a config redeploy.
Keeps the umbrella index in step with the OtOpcUa driver set, per the
propagation rule in that repo's CLAUDE.md. Read-only Agent driver:
hand-rolled HTTP/XML client (probe/current + multipart /sample pump),
TrakHound dependency evaluated and dropped, browse free via the Wave-0
universal discovery browser, write-back deferred. Live-gated against a
real mtconnect/agent 2.7.0.12 fixture.
Sister projects are told to update their entry here whenever an indexed fact
changes, but that edit lands in this repo's own checkout — so a naive commit
writes it onto whatever branch scadaproj happens to be on. Observed 2026-07-27:
the OtOpcUa Sql driver's entry sat unpushed on feat/overview-dashboard for three
days, and the Mqtt entry was about to join it.
Records the procedure (stash, main, push, restore) and says to cherry-pick
stranded index commits rather than merge the branch they are stuck on.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The OtOpcUa entry's driver set changed, so the index has to change with it
(OtOpcUa CLAUDE.md's cross-repo rule).
A read-only `Mqtt` broker-subscribe driver landed with two ingest shapes over
one MQTTnet-5 client: Plain binds a tag to a concrete topic, SparkplugB decodes
vendored Eclipse-Tahu protobuf under a single spBv1.0/{groupId}/# subscription
and binds by the group/edgeNode/device?/metric tuple (birth/alias/seq-gap/
rebirth state machine, death→STALE, a birth-driven browse picker with a scoped
Request-rebirth NCMD). Write-through is deferred — every MQTT node is read-only.
Live-gated on docker-dev against a Mosquitto TLS+auth broker plus a
project-owned C# Sparkplug edge-node simulator on 10.100.0.35:8883.
Recorded as **on branch feat/mqtt-sparkplug-driver, not yet merged to master**,
so the index does not overstate what is on OtOpcUa's default branch — unlike the
Sql DB-poll driver next to it, which is merged.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
OtOpcUa merged a read-only Sql Equipment-kind driver to master 2026-07-24
(polls SQL Server → OPC UA variables; connectionStringRef env resolution,
schema-walk address picker, live /run gate passed on docker-dev). Keep the
umbrella index's OtOpcUa driver enumeration in sync per the propagation rule.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The OtOpcUa entry's driver set changed, so the index has to change with it
(OtOpcUa CLAUDE.md's cross-repo rule).
A read-only `Mqtt` broker-subscribe driver landed with two ingest shapes over
one MQTTnet-5 client: Plain binds a tag to a concrete topic, SparkplugB decodes
vendored Eclipse-Tahu protobuf under a single spBv1.0/{groupId}/# subscription
and binds by the group/edgeNode/device?/metric tuple (birth/alias/seq-gap/
rebirth state machine, death→STALE, a birth-driven browse picker with a scoped
Request-rebirth NCMD). Write-through is deferred — every MQTT node is read-only.
Live-gated on docker-dev against a Mosquitto TLS+auth broker plus a
project-owned C# Sparkplug edge-node simulator on 10.100.0.35:8883.
Recorded as **on branch feat/mqtt-sparkplug-driver, not yet merged to master**,
so the index does not overstate what is on OtOpcUa's default branch — unlike the
Sql DB-poll driver next to it, which is merged.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
OtOpcUa merged a read-only Sql Equipment-kind driver to master 2026-07-24
(polls SQL Server → OPC UA variables; connectionStringRef env resolution,
schema-walk address picker, live /run gate passed on docker-dev). Keep the
umbrella index's OtOpcUa driver enumeration in sync per the propagation rule.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The dashboard derived every leader chip from AkkaClusterHealthCheck's `leader`
data — ClusterState.Leader, the LOWEST-ADDRESSED Up member. That is not the node
in charge. The node holding the singletons, gated for writes, and answering 200
on /health/active is the OLDEST Up member. The two agree only until some node
restarts, after which the restarted node rejoins youngest but keeps its address.
Observed live on the rebuilt OtOpcUa rig during the Health 0.3.0 work: leader
central-1, active node central-2. The dashboard would have put the Active chip on
central-1 while /health/active said central-2 — the chip contradicting the tier
it exists to visualise. The Phase 4 acceptance run missed it because the rig had
just been formed, the one state in which the two orderings agree.
Health 0.3.0's active tier now publishes `activeNode` on every member, including
standbys, so the fix is to prefer it and keep `leader` as the fallback for an
instance on an older Health or whose active tier was not probed.
The instance card now shows `active` and `leader` side by side rather than
swapping one for the other: they are legitimately different nodes, and showing
both turns a silent contradiction into something readable at 3am.
Tests: 177 (+3) — precedence when the two disagree, a standby naming the active
node rather than itself, and the fallback still working with no activeNode.
Rewrites the active-node sections of the spec and shared contract, which
described the leader-based design that both consumers rejected.
- SPEC §2.3 replaced: the rule, the ordered RolePreference, why a node holding
none of the preferred roles is Unhealthy rather than Healthy, and the data
payload. §4 (IActiveNodeGate) and the tier table follow.
- SPEC §6 marked historical: it still tells adopters to use
ActiveNodeHealthCheck(role: "admin") and AkkaActiveNodeGate, which is the
advice that produced the defect. Kept for provenance, annotated so nobody
follows it again.
- GAPS: gaps L1/L2 recorded as closed the wrong way, reopened, then closed at
0.3.0 — including the part worth remembering, that both apps adopted the
shared check as specified, hit it live, and reverted to private copies, so the
surface reached zero consumers by being avoided rather than unused.
- Index rows updated with the consumer-visible changes and both app branches.
Promotes the two apps' private replacements for ActiveNodeHealthCheck into one
shared primitive, and retires the leader/RoleLeader selection they were written
to avoid.
Both Akka consumers had already hand-rolled a replacement rather than use this
package (ScadaBridge OldestNodeActiveHealthCheck, OtOpcUa
ClusterPrimaryHealthCheck), so the entire active-node surface here —
ActiveNodeHealthCheck, AkkaActiveNodeGate — had ZERO consumers. It was not
merely unused: it was avoided, twice, for the same reason.
Leadership is address-ordered (host, then port) and has no relationship to time;
singleton placement is age-ordered. The two agree on a freshly-formed cluster,
which is why single-node and happy-path tests never caught it. They diverge
permanently after any restart: the restarted node rejoins as the youngest but
keeps its address, so if it holds the lower address it becomes leader while the
singletons — and all the work they own — stay on the other node.
New ClusterActiveNode is the single implementation: oldest Up member, optional
role scope, plus role-preference resolution for a fused node that must answer
for the role its singletons are pinned to. ActiveNodeHealthCheck and
AkkaActiveNodeGate both delegate to it, so an endpoint gate and the
/health/active probe an orchestrator routes by cannot disagree.
BREAKING (behaviour, not signature):
- Selection is by age, not leadership.
- The role-filtered mode no longer reports Healthy for a node LACKING the role.
That "not applicable => Healthy" made the tier answer 200 on every node and
silently broke leader-pinning (lmxopcua#494); it is now Unhealthy by default,
overridable via NoActiveRoleStatus. This case is reachable, not defensive —
OtOpcUa's RoleParser admits dev-only and cluster-role-only nodes.
- Identity compares UniqueAddress, so a node restarted on the same host:port is
correctly a different member during the overlap.
Results now carry activeRole/selfAddress/activeNode in the 0.2.0 per-entry data
object, so a standby reports WHO is active and a dashboard can render a pair
from either node's payload alone.
Tests: 45 Akka (was 39), 76 total. The age-vs-address divergence is pinned
against a real two-node cluster built so the oldest member is not the
lowest-addressed one, with a fixture sanity check so it cannot pass for the
wrong reason.
The shared spec states the active tier "Fails (503) on a standby or
role-member-but-not-leader node", and the same spec maps Degraded to HTTP 200.
ActiveNodeDecision.Evaluate returned Degraded for exactly that case, so the tier
answered 200 on every node and could not distinguish an active node from a
standby over HTTP at all - only the response body differed.
The consequence was live, not theoretical: OtOpcUa's Traefik routes the admin UI
by this probe (docker-dev/traefik-dynamic.yml, scripts/install/traefik-dynamic.yml),
so both central nodes always passed and the leader-pinning the design called for
has silently never worked. Found while running the overview dashboard's live
acceptance, which reads the same tier and showed all six rig nodes as Active.
Filed as lmxopcua#494.
Role-member-but-not-leader now returns Unhealthy. The other two branches are
unchanged: a node that lacks the role stays Healthy (the probe is irrelevant to
it), and the startup-safety path still returns Degraded when the ActorSystem or
cluster is not yet available.
Blast radius is one consumer. OtOpcUa is the only user of the role-filtered mode;
ScadaBridge deliberately uses its own OldestNodeActiveHealthCheck (leadership is
address-ordered and diverges from the singleton host), and MxGateway and
HistorianGateway have no Akka cluster.
Version 0.2.1, three packages published to the Gitea feed and restore-verified.
70 tests pass.
Found by the live rig, and invisible to every unit test that existed.
LeaderResolver and InstanceCard.ClusterDataLine both looked the cluster view up
under the check name "akka-cluster". ScadaBridge registers the shared
AkkaClusterHealthCheck under that name; OtOpcUa registers the SAME check as
"akka". So once the OtOpcUa rig was rebuilt on Health 0.2.0 and started
publishing data.leader, its groups still rendered with no leader chip and no
evidence line - and a group with no votes is a legitimate state, so nothing
looked wrong. ScadaBridge's four groups were fully populated the whole time,
which is what made it look like the feature worked.
Both now find the entry by the DATA KEY ("leader"), which is the part of the
contract the shared check actually owns rather than the part each consumer names
for itself.
Regression tests: the leader is found under akka-cluster, akka, and an arbitrary
future name; and an entry that publishes data WITHOUT a leader (ScadaBridge's
localdb check, whose data carries replication counters) does not get picked in
preference to the one that answers the question. Every prior fixture used
ScadaBridge's name, which is precisely why this got through.
After the fix all seven cluster groups render a leader chip with zero
split-brain flags - design section 9 check 2 now passes for BOTH products.
174 tests, 0 warnings.
Found during the live acceptance run. The host calls UseStaticWebAssets only when
the environment is literally "Development", so running the build output under any
other name - checking a staging registry locally, say - leaves the Theme RCL's
assets under _content/ unresolvable. The failure is silent and unhelpful: the page
returns 200 with its markup fully present and renders as a blank white screen,
because every stylesheet 404s. Calling it unconditionally is a no-op once
published (the manifest is gone and the assets are real files on disk), so the
only thing it changes is that failure. The container was never affected.
Phase 4 results recorded in the ledger: 7 of 9 design section 9 checks fully pass,
2 pass for ScadaBridge and are blocked for OtOpcUa pending a rig redeploy.
Notable live evidence:
- Leader chip moved from site-c-a to site-c-b when site-c-a was stopped, and the
split-brain chip appeared while the pair was self-formed then cleared once it
genuinely joined.
- Kill to Unreachable took two failing sweeps (the damping working as designed);
recovery landed on the first good poll.
- Five consecutive page loads in 0.00-0.02s all showed the same "last sweep"
timestamp, which is cache-first stated as a measurement.
- Freezing the probed endpoints while auto-refresh stayed RUNNING froze the poller
mid-sweep and the cards aged into STALE on their own - staleness is computed at
render, so a stuck poller needs no explicit stall detection.
The run also surfaced a pre-existing rig defect in both products: self-first Akka
seed lists mean a pair booted cold self-forms two 1-member clusters that never
join. After a --force-recreate every ScadaBridge node named ITSELF leader and every
node answered /health/active 200; restarting ONE node of each pair converged them
to one Leader plus one Active plus one Standby. That is the user's own open backlog
item (docs/plans/2026-07-22-initjoin-selfform-fallback.md, unexecuted), and the
dashboard made it visible fleet-wide in one screen.
170 tests, 0 warnings.
Closes Phase 3 of the overview-dashboard plan (tasks 3.7-3.9).
Self-observability
- OverviewMetrics: an observable gauge read from the published snapshot at scrape
time, so the metric and the page cannot disagree about what the dashboard
believes, plus a sweep-duration histogram. Instrument names follow the family
METRIC-CONVENTIONS spec (overview.instance.status / overview.poll.duration,
unit s) rather than the plan's literal zb_ names; Prometheus renders them as
overview_instance_status and overview_poll_duration_seconds. Labels are
application/node/group - "node", not "instance", because Prometheus stamps its
own instance label on every scraped series and a colliding metric label is
silently renamed exported_instance. Instances never probed emit nothing: the
enum's zero ordinal is Up, so a placeholder would publish the single most
reassuring answer about a node nobody has heard from.
- Two readiness checks make the dashboard's own /health/ready mean something:
registry-loaded (reports the counts it bound) and last-poll-cycle-completed,
which catches the failure where Kestrel is fine, the page still renders, and
every card on it is quietly frozen. The heartbeat is stamped by the sweep, not
derived from the snapshot's timestamp - the poller publishes the registry
before it probes anything, so a stuck poller would otherwise look healthy.
No cycle yet is Degraded inside the grace window and Unhealthy past it, so a
restart does not fail readiness for its first seconds and a poller that never
started cannot hide behind "still starting".
Configuration
- appsettings.json ships tuning defaults with an EMPTY Applications list, so a
deployment that supplies no registry fails at ConfigPreflight naming the key
instead of booting into a dashboard of nothing.
- Three registries, because reachability genuinely differs: Development is the
host-reachable subset (4 instances), Docker is the full 16-instance fleet by
container DNS name. The rigs publish almost no per-node HTTP port to the host
- OtOpcUa publishes none (only Traefik :9200, load-balanced across the central
pair, so it cannot identify a node) and ScadaBridge keeps site :8084
container-internal. Ports were established by probing the running rigs, not
read off the compose files, because the two disagree: OtOpcUa site nodes serve
health on :8080, not the :9000 their compose anchor implies.
- Real Serilog sinks, not the family's empty "Serilog": {}. AddZbSerilog drives
sinks entirely from configuration and adds none of its own, so the empty
section produces an app that logs NOWHERE - verified against the running
HistorianGateway container, whose log stream is completely empty for exactly
this reason. The HttpClient level override matters too: 16 instances on a 10s
cadence emit several Information lines a second of pure probe noise.
Container
- HistorianGateway-pattern runtime-only image; the compose stack joins the three
rig networks as external, so bringing the dashboard up or down cannot disturb
a rig. The Dockerfile clears ASPNETCORE_HTTP_PORTS as well as ASPNETCORE_URLS:
on .NET 8+ the base image declares its binding via the former, and clearing
only the latter leaves Kestrel logging "Overriding address(es)" every boot.
No healthcheck - aspnet:10.0 has neither curl nor wget, and adding a
--healthcheck entry point to production code to satisfy Compose is not a
trade worth making.
Tests
- BootTests boot the real Program.cs: every endpoint anonymous, the page
rendering from cache while nothing it watches is reachable, the Meters
allowlist proven by an end-to-end /metrics scrape, and both refuse-to-boot
paths (missing registry, stale window inside the poll interval).
The registry is supplied via environment variables, not
ConfigureAppConfiguration, because ConfigPreflight runs before Build() - when
the factory's callbacks are applied - and would not see them.
170 tests, 0 warnings. Verified live against the running rigs: 16 cards, 16
gauge series, page and gauge agreeing exactly.
Tasks 3.4-3.6 of the overview-dashboard plan.
Polling (3.4/3.5):
- OverviewPollerService: one timer at the fastest configured cadence, each
target polled when its own interval is due, fan-out per sweep, no retries
(the next tick is the retry, so damping stays the only place that decides
whether a failure is worth showing).
- OverviewSnapshotStore: atomic swap of a complete immutable graph plus a
change event. Pages read it and never probe — the cache-first requirement.
- LeaderResolver: per-group leader from akka-cluster data, with the
split-brain flag scoped to members that are actually answering, so an
ordinary failover is not misread as a disagreement.
UI (3.6): ThemeShell + StatusPill/TechCard composition against the mockup;
instance cards carry the status stripe (dashed for Unreachable), the check
detail list and the raw ready/active signal line.
Two defects found by verification rather than by review:
- The active tier answers with a status code and an EMPTY body, but the client
demanded parseable JSON on every probe — so a healthy pair rendered as
"role unknown" instead of Active/Standby. Split into ProbeAsync (ready tier,
body is the payload) and ProbeStatusAsync (active tier, code is the answer).
Caught by a live smoke run against two fake health endpoints; the poller
tests now serve an empty body so they hold the fix.
- SweepAsync published even when cancellation was already requested, leaving a
half-probed registry as the store's final state on shutdown. It now checks
the token itself rather than relying on the transport to throw.
Also: app.UseAntiforgery() is required despite the anonymous-by-design
pipeline — AddRazorComponents stamps antiforgery metadata unconditionally and
the endpoint middleware hard-fails without it (every page was 500). Chosen
over DisableAntiforgery() so a future form is protected by default.
147 tests, 0 warnings. Live-verified end to end against fake endpoints:
Up/Unreachable/Active/Standby, leader chip, cluster-data line, KPI counts.
Phase 3 tasks 3.1-3.3 of docs/plans/2026-07-22-overview-dashboard-impl-plan.md.
Scaffold (3.1): a plain directory in scadaproj (NOT a nested git repo), slnx over
src/ + tests/, HistorianGateway-pattern Directory.Build.props (warnings-as-errors)
and nuget.config (nuget.org * + the Gitea feed scoped to ZB.MOM.WW.*). Inline
package pins, no CPM — the app convention here. NO Auth/Audit/Secrets packages:
the dashboard is anonymous and read-only by requirement.
Registry (3.2): OverviewOptions/ApplicationEntry/InstanceEntry bound from the
"Overview" section, plus EffectiveTimings resolving instance > application >
global overrides in ONE place so no caller re-implements the precedence.
OverviewOptionsValidator (OptionsValidatorBase + AddValidatedOptions) rejects an
empty registry, applications with no instances, duplicate names, non-absolute or
non-http(s) URLs, and non-positive intervals — and checks stale > poll on the
EFFECTIVE values, since an override at either level can invert that on one
instance while the globals look fine. A ConfigPreflight presence check runs before
the host is built so a missing registry fails with the key name, not with
"Applications must have at least 1 entry".
Health client (3.3): ZbHealthReport DTOs for the canonical ZbHealthWriter body,
with per-entry `data` kept as JsonElement — it is an open contract, and binding it
to concrete types would break the moment a check adds a field. 200 AND 503 are
both parseable answers (503 carries the body naming the failing check); only a
transport failure or a non-canonical body is a failed probe, which is why
Unreachable stays distinct from Down. Host shutdown propagates as cancellation
rather than being recorded as every instance timing out.
Status model: Up/Degraded/Down/Unreachable + Active/Standby/Unknown/NotApplicable,
with two-strike flap damping — a failing observation only replaces a good state
after 2 consecutive failures, while recovery is immediate (damping suppresses
false alarms; symmetric damping would just make them linger).
53 tests green, 0 warnings in Debug and Release. Tests for the validator, the
client golden payloads (with AND without `data`, so a partly-bumped fleet renders)
and the derivation table are front-loaded here from task 3.8 so this batch is
verified rather than pending.
Also pins AngleSharp 1.5.2 test-side: bunit 1.40.0 pulls 1.2.0, which trips NU1902
and therefore the warnings-as-errors gate (same fix HistorianGateway made in
6bc005d).
Phase 0 of docs/plans/2026-07-22-overview-dashboard-impl-plan.md: give the
canonical health JSON a structured channel so the family overview dashboard can
read each Akka cluster's current leader.
- ZbHealthWriter: optional `"data": {...}` per entry, sourced from
HealthReportEntry.Data, emitted only when non-empty. Per-property JsonIgnore
(NOT a global DefaultIgnoreCondition) so `"description": null` still renders —
payloads from data-less checks stay byte-identical to 0.1.0.
- AkkaClusterHealthCheck: BuildClusterData publishes this node's own view —
leader (omitted while unknown), selfAddress, selfRoles (sorted), memberCount,
unreachableCount — on every result path. The startup-safety paths (no
ActorSystem / cluster inaccessible) stay description-only.
- Tests: writer data emit/omit (raw-JSON assert on the omit case), and a real
single-node self-joined cluster via Akka.TestKit.Xunit2 for the data values.
70 tests green (25/39/6).
- Version 0.1.0 -> 0.2.0; 3 packages published to the Gitea feed and
restore-verified from a scratch consumer, which serves data.leader live.
Per-cluster mesh Phase 6 merged to OtOpcUa master 2839deb5 (pushed 2026-07-24).
Records the split into one 2-node mesh per application Cluster, the consumer-visible
per-Cluster ServiceLevel/Primary scoping (two+ Primaries fleet-wide by design, no
wire-contract change), the SplitTopologyTransportValidator, pair-local secrets, and
the deferred live gate.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 4 merged + pushed (tip 3a590a0c): driver-only nodes run with no ConfigDb
connection string. Consumer-visible ServiceLevel change — a DB-less site node
now holds 240/250 with central SQL down (survive-alone), where a DB-backed node
would drop to 0/100. Phases 0a/0b/1/2/3/4 done; 5-7 not started.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Update the OtOpcUa entry: the per-cluster mesh election-scoping fix now has Phases
0a/0b/1/2/3 merged + pushed (tip d01b0695). Phase 2 = ClusterClient command transport
(MeshTransport:Mode dark switch). Phase 3 = config fetch-and-cache — a FetchAndCache
driver node boots + serves last-known-good with central SQL down (extends the #485
availability guarantee consumers like ScadaBridge rely on); a failed fetch is 'no answer',
never an empty config. Live gate caught + fixed a Status==Sealed serve-gate deadlock.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Options investigation (Aspire standalone/HealthChecks.UI/Gatus/Homepage/
Grafana vs custom) → decision: custom ZB.MOM.WW.Overview Blazor app in the
family look. Design: anonymous read-only single pane, appsettings registry
of all four apps' instances, cache-first poller on a configurable timer,
configurable staleness timeout, Active/Standby from /health/active, and
per-cluster Akka leader chips (needs Health 0.2.0 optional per-entry data).
Impl plan: phased + code-verified — Phase 0 Health 0.2.0 (writer data field
+ AkkaClusterHealthCheck cluster view), Phase 1 ScadaBridge site-node
MapZbHealth PR (:8084; role-scoped active check), Phase 2 OtOpcUa bump,
Phase 3 the app (:5320), Phase 4 live rig acceptance. Mockup is the visual
reference: real Theme tokens + embedded IBM Plex, all card states incl.
stale + split-brain. Ready to execute in a fresh session.
Corrects the same-day report: PR searches for 'quic' missed work named
'artery'. Artery is landing rapidly on upstream dev as Artery-TCP (openspec
#8313; framing/handshake G1-G2; parallel lanes #8356/#8357; large-message
streamId 3 #8352; MNTR/CI for both transports), with Serialization.V2 in
parallel. TCP-first resolves §7's forward note: pipeline benefits arrive
without libmsquic, dropping the §6 platform constraints from the 1.6
adoption path. No previews or public benchmarks yet.
Two research reports (2026-07-22):
- akka_msquic_transport.md: what a QUIC transport on Akka 1.5.62 would take,
the 1.5-vs-1.6 pipeline delta that gates the benefits, .NET-native
alternatives (managed sockets ranked first), and measured throughput
evidence (PR #4594: transport swap is throughput-neutral).
- akka_multidc_clusters.md: JVM multi-DC semantics mapped onto the
central-pair + site-pairs topology, the stalled Akka.NET port (#3261 /
PR #4111), per-system fit vs the OtOpcUa mesh program, and the
before-Phase-2 decision point.
Recommendation in both: don't build; track upstream.
Both are consumer-visible, so they belong in the index rather than only in the
repo. The downing strategy moved to AutoDowning because a two-node OtOpcUa pair
on keep-oldest could not survive a crash of the oldest node — the same defect and
the same fix as ScadaBridge cf3bd52f, which is where the analysis came from. And
the redundancy Primary is now the oldest Up driver member rather than the role
leader, which changes which node advertises ServiceLevel 250 after a restart —
relevant to anything selecting a server by ServiceLevel.
Also notes that the four NU1903 advisories which broke ScadaBridge's fresh
restores were breaking OtOpcUa's too, and that the election remains scoped per
Akka cluster until the per-cluster mesh work lands.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The #485/#486 fixes changed nothing structural about OtOpcUa — same drivers,
same dual-namespace scheme, same Galaxy flow — so the row is mostly untouched.
What DID change is a fact any OPC UA consumer depends on, and ScadaBridge is
mid-cutover onto exactly this address space: a transient ConfigDb error used to
empty the served address space (a PureRemove of every node, observed live and it
stayed empty), stop every driver and clear every subscription, while still
reporting the deploy Applied. A consumer would have seen all bindings go bad
with nothing upstream admitting a problem. Recorded on the OtOpcUa row.
Also fixes real drift found while checking: the OtOpcUa LocalDb adoption design
was sitting UNTRACKED here and unreferenced by the index, even though the
LocalDb row claims the adoption designs live in this repo. Now committed,
referenced alongside the ScadaBridge one, and its stale header corrected —
it still said Phase 1 was "not yet executed" against lib 0.1.1, when Phase 1
shipped and merged against 0.1.3. Phase 2 designed but not executed, on both
the doc and the row.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Two defects about a node whose LocalDb file is lost, the second only reachable
once the first was fixed:
- 0.1.2 — a converged pair prunes its oplog to empty on ack, and snapshot
detection read an empty oplog as 'no gap possible'. The steady state of a
healthy pair was the one state from which a rebuilt node could never be
back-filled.
- 0.1.3 — with back-fill working, the rebuilt node's OWN writes were silently
dropped: last_applied_remote_seq is a watermark in the peer's seq space and a
rebuilt peer numbers from 1, so the healthy node's stale watermark made the
sending side skip its whole oplog.
Both found by the OtOpcUa LocalDb Phase 1 live gate on the docker-dev rig and
reproduced offline as RED tests first. 149/149 pass.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The other half of the rebuilt-peer story, and only reachable once 0.1.2 made
back-fill work: with the wiped node repopulated and writing again, its writes
never reached its peer.
last_applied_remote_seq is a watermark in the PEER's seq space, and a rebuilt
peer is a new database whose oplog numbers from 1. Two things then went wrong,
and both had to be fixed — the first alone leaves data stuck:
- The healthy node kept advertising the OLD peer's watermark, and the sending
side seeds its pump from exactly that number, so the rebuilt node skipped its
entire oplog. The watermark (and the observed peer clock) is now reset when
the peer's node id changes. last_acked_seq is deliberately NOT reset: it
describes our own oplog's pruned horizon and is what tells a rebuilt peer it
needs a snapshot — clearing it would turn 0.1.2's back-fill back off.
- A peer claiming to have applied more of our stream than we have ever produced
can only be remembering a previous incarnation of us. That claim is now
rejected in favour of starting from the beginning; LWW makes the re-send
harmless. This mirrors the clamp RecordPeerAckAsync already applies to acks.
Found on the OtOpcUa docker-dev rig, where the healthy node held watermark 10
while the rebuilt peer's oplog was seqs 1-3 and its last_acked stayed 0 — the
peer never accepted a single one. Covered by a test that converges with one
peer, replaces it with a fresh database, and requires a write on the new peer
to arrive: RED before this fix, green after. 149/149 pass.
Version 0.1.3.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Joseph Doherty