Commit Graph

380 Commits

Author SHA1 Message Date
Joseph Doherty af1e0a86b2 fix(health): active tier returns 503 on a role-member-but-not-leader node (0.2.1)
The shared spec states the active tier "Fails (503) on a standby or
role-member-but-not-leader node", and the same spec maps Degraded to HTTP 200.
ActiveNodeDecision.Evaluate returned Degraded for exactly that case, so the tier
answered 200 on every node and could not distinguish an active node from a
standby over HTTP at all - only the response body differed.

The consequence was live, not theoretical: OtOpcUa's Traefik routes the admin UI
by this probe (docker-dev/traefik-dynamic.yml, scripts/install/traefik-dynamic.yml),
so both central nodes always passed and the leader-pinning the design called for
has silently never worked. Found while running the overview dashboard's live
acceptance, which reads the same tier and showed all six rig nodes as Active.
Filed as lmxopcua#494.

Role-member-but-not-leader now returns Unhealthy. The other two branches are
unchanged: a node that lacks the role stays Healthy (the probe is irrelevant to
it), and the startup-safety path still returns Degraded when the ActorSystem or
cluster is not yet available.

Blast radius is one consumer. OtOpcUa is the only user of the role-filtered mode;
ScadaBridge deliberately uses its own OldestNodeActiveHealthCheck (leadership is
address-ordered and diverges from the singleton host), and MxGateway and
HistorianGateway have no Akka cluster.

Version 0.2.1, three packages published to the Gitea feed and restore-verified.
70 tests pass.
2026-07-24 10:46:04 -04:00
Joseph Doherty c22b06deba docs(overview): link the OtOpcUa active-tier finding to lmxopcua#494 2026-07-24 10:40:45 -04:00
Joseph Doherty d95292c95d fix(overview): locate the cluster view by data key, not by check name
Found by the live rig, and invisible to every unit test that existed.

LeaderResolver and InstanceCard.ClusterDataLine both looked the cluster view up
under the check name "akka-cluster". ScadaBridge registers the shared
AkkaClusterHealthCheck under that name; OtOpcUa registers the SAME check as
"akka". So once the OtOpcUa rig was rebuilt on Health 0.2.0 and started
publishing data.leader, its groups still rendered with no leader chip and no
evidence line - and a group with no votes is a legitimate state, so nothing
looked wrong. ScadaBridge's four groups were fully populated the whole time,
which is what made it look like the feature worked.

Both now find the entry by the DATA KEY ("leader"), which is the part of the
contract the shared check actually owns rather than the part each consumer names
for itself.

Regression tests: the leader is found under akka-cluster, akka, and an arbitrary
future name; and an entry that publishes data WITHOUT a leader (ScadaBridge's
localdb check, whose data carries replication counters) does not get picked in
preference to the one that answers the question. Every prior fixture used
ScadaBridge's name, which is precisely why this got through.

After the fix all seven cluster groups render a leader chip with zero
split-brain flags - design section 9 check 2 now passes for BOTH products.

174 tests, 0 warnings.
2026-07-24 10:38:39 -04:00
Joseph Doherty 41013dcab3 fix(overview): call UseStaticWebAssets unconditionally; record Phase 4 results
Found during the live acceptance run. The host calls UseStaticWebAssets only when
the environment is literally "Development", so running the build output under any
other name - checking a staging registry locally, say - leaves the Theme RCL's
assets under _content/ unresolvable. The failure is silent and unhelpful: the page
returns 200 with its markup fully present and renders as a blank white screen,
because every stylesheet 404s. Calling it unconditionally is a no-op once
published (the manifest is gone and the assets are real files on disk), so the
only thing it changes is that failure. The container was never affected.

Phase 4 results recorded in the ledger: 7 of 9 design section 9 checks fully pass,
2 pass for ScadaBridge and are blocked for OtOpcUa pending a rig redeploy.

Notable live evidence:
- Leader chip moved from site-c-a to site-c-b when site-c-a was stopped, and the
  split-brain chip appeared while the pair was self-formed then cleared once it
  genuinely joined.
- Kill to Unreachable took two failing sweeps (the damping working as designed);
  recovery landed on the first good poll.
- Five consecutive page loads in 0.00-0.02s all showed the same "last sweep"
  timestamp, which is cache-first stated as a measurement.
- Freezing the probed endpoints while auto-refresh stayed RUNNING froze the poller
  mid-sweep and the cards aged into STALE on their own - staleness is computed at
  render, so a stuck poller needs no explicit stall detection.

The run also surfaced a pre-existing rig defect in both products: self-first Akka
seed lists mean a pair booted cold self-forms two 1-member clusters that never
join. After a --force-recreate every ScadaBridge node named ITSELF leader and every
node answered /health/active 200; restarting ONE node of each pair converged them
to one Leader plus one Active plus one Standby. That is the user's own open backlog
item (docs/plans/2026-07-22-initjoin-selfform-fallback.md, unexecuted), and the
dashboard made it visible fleet-wide in one screen.

170 tests, 0 warnings.
2026-07-24 08:19:41 -04:00
Joseph Doherty 92997b40dc feat(overview): self-observability, config and the container image
Closes Phase 3 of the overview-dashboard plan (tasks 3.7-3.9).

Self-observability
- OverviewMetrics: an observable gauge read from the published snapshot at scrape
  time, so the metric and the page cannot disagree about what the dashboard
  believes, plus a sweep-duration histogram. Instrument names follow the family
  METRIC-CONVENTIONS spec (overview.instance.status / overview.poll.duration,
  unit s) rather than the plan's literal zb_ names; Prometheus renders them as
  overview_instance_status and overview_poll_duration_seconds. Labels are
  application/node/group - "node", not "instance", because Prometheus stamps its
  own instance label on every scraped series and a colliding metric label is
  silently renamed exported_instance. Instances never probed emit nothing: the
  enum's zero ordinal is Up, so a placeholder would publish the single most
  reassuring answer about a node nobody has heard from.
- Two readiness checks make the dashboard's own /health/ready mean something:
  registry-loaded (reports the counts it bound) and last-poll-cycle-completed,
  which catches the failure where Kestrel is fine, the page still renders, and
  every card on it is quietly frozen. The heartbeat is stamped by the sweep, not
  derived from the snapshot's timestamp - the poller publishes the registry
  before it probes anything, so a stuck poller would otherwise look healthy.
  No cycle yet is Degraded inside the grace window and Unhealthy past it, so a
  restart does not fail readiness for its first seconds and a poller that never
  started cannot hide behind "still starting".

Configuration
- appsettings.json ships tuning defaults with an EMPTY Applications list, so a
  deployment that supplies no registry fails at ConfigPreflight naming the key
  instead of booting into a dashboard of nothing.
- Three registries, because reachability genuinely differs: Development is the
  host-reachable subset (4 instances), Docker is the full 16-instance fleet by
  container DNS name. The rigs publish almost no per-node HTTP port to the host
  - OtOpcUa publishes none (only Traefik :9200, load-balanced across the central
  pair, so it cannot identify a node) and ScadaBridge keeps site :8084
  container-internal. Ports were established by probing the running rigs, not
  read off the compose files, because the two disagree: OtOpcUa site nodes serve
  health on :8080, not the :9000 their compose anchor implies.
- Real Serilog sinks, not the family's empty "Serilog": {}. AddZbSerilog drives
  sinks entirely from configuration and adds none of its own, so the empty
  section produces an app that logs NOWHERE - verified against the running
  HistorianGateway container, whose log stream is completely empty for exactly
  this reason. The HttpClient level override matters too: 16 instances on a 10s
  cadence emit several Information lines a second of pure probe noise.

Container
- HistorianGateway-pattern runtime-only image; the compose stack joins the three
  rig networks as external, so bringing the dashboard up or down cannot disturb
  a rig. The Dockerfile clears ASPNETCORE_HTTP_PORTS as well as ASPNETCORE_URLS:
  on .NET 8+ the base image declares its binding via the former, and clearing
  only the latter leaves Kestrel logging "Overriding address(es)" every boot.
  No healthcheck - aspnet:10.0 has neither curl nor wget, and adding a
  --healthcheck entry point to production code to satisfy Compose is not a
  trade worth making.

Tests
- BootTests boot the real Program.cs: every endpoint anonymous, the page
  rendering from cache while nothing it watches is reachable, the Meters
  allowlist proven by an end-to-end /metrics scrape, and both refuse-to-boot
  paths (missing registry, stale window inside the poll interval).
  The registry is supplied via environment variables, not
  ConfigureAppConfiguration, because ConfigPreflight runs before Build() - when
  the factory's callbacks are applied - and would not see them.

170 tests, 0 warnings. Verified live against the running rigs: 16 cards, 16
gauge series, page and gauge agreeing exactly.
2026-07-24 07:35:32 -04:00
Joseph Doherty 5fe7135e2c feat(overview): poller, leader aggregation and the dashboard UI
Tasks 3.4-3.6 of the overview-dashboard plan.

Polling (3.4/3.5):
- OverviewPollerService: one timer at the fastest configured cadence, each
  target polled when its own interval is due, fan-out per sweep, no retries
  (the next tick is the retry, so damping stays the only place that decides
  whether a failure is worth showing).
- OverviewSnapshotStore: atomic swap of a complete immutable graph plus a
  change event. Pages read it and never probe — the cache-first requirement.
- LeaderResolver: per-group leader from akka-cluster data, with the
  split-brain flag scoped to members that are actually answering, so an
  ordinary failover is not misread as a disagreement.

UI (3.6): ThemeShell + StatusPill/TechCard composition against the mockup;
instance cards carry the status stripe (dashed for Unreachable), the check
detail list and the raw ready/active signal line.

Two defects found by verification rather than by review:

- The active tier answers with a status code and an EMPTY body, but the client
  demanded parseable JSON on every probe — so a healthy pair rendered as
  "role unknown" instead of Active/Standby. Split into ProbeAsync (ready tier,
  body is the payload) and ProbeStatusAsync (active tier, code is the answer).
  Caught by a live smoke run against two fake health endpoints; the poller
  tests now serve an empty body so they hold the fix.

- SweepAsync published even when cancellation was already requested, leaving a
  half-probed registry as the store's final state on shutdown. It now checks
  the token itself rather than relying on the transport to throw.

Also: app.UseAntiforgery() is required despite the anonymous-by-design
pipeline — AddRazorComponents stamps antiforgery metadata unconditionally and
the endpoint middleware hard-fails without it (every page was 500). Chosen
over DisableAntiforgery() so a future form is protected by default.

147 tests, 0 warnings. Live-verified end to end against fake endpoints:
Up/Unreachable/Active/Standby, leader chip, cluster-data line, KPI counts.
2026-07-24 07:05:16 -04:00
Joseph Doherty 61fc88c8a8 chore(plan): overview dashboard task ledger through task 3.3 2026-07-24 06:42:28 -04:00
Joseph Doherty d0110cf406 feat(overview): scaffold the family dashboard + registry + health client
Phase 3 tasks 3.1-3.3 of docs/plans/2026-07-22-overview-dashboard-impl-plan.md.

Scaffold (3.1): a plain directory in scadaproj (NOT a nested git repo), slnx over
src/ + tests/, HistorianGateway-pattern Directory.Build.props (warnings-as-errors)
and nuget.config (nuget.org * + the Gitea feed scoped to ZB.MOM.WW.*). Inline
package pins, no CPM — the app convention here. NO Auth/Audit/Secrets packages:
the dashboard is anonymous and read-only by requirement.

Registry (3.2): OverviewOptions/ApplicationEntry/InstanceEntry bound from the
"Overview" section, plus EffectiveTimings resolving instance > application >
global overrides in ONE place so no caller re-implements the precedence.
OverviewOptionsValidator (OptionsValidatorBase + AddValidatedOptions) rejects an
empty registry, applications with no instances, duplicate names, non-absolute or
non-http(s) URLs, and non-positive intervals — and checks stale > poll on the
EFFECTIVE values, since an override at either level can invert that on one
instance while the globals look fine. A ConfigPreflight presence check runs before
the host is built so a missing registry fails with the key name, not with
"Applications must have at least 1 entry".

Health client (3.3): ZbHealthReport DTOs for the canonical ZbHealthWriter body,
with per-entry `data` kept as JsonElement — it is an open contract, and binding it
to concrete types would break the moment a check adds a field. 200 AND 503 are
both parseable answers (503 carries the body naming the failing check); only a
transport failure or a non-canonical body is a failed probe, which is why
Unreachable stays distinct from Down. Host shutdown propagates as cancellation
rather than being recorded as every instance timing out.

Status model: Up/Degraded/Down/Unreachable + Active/Standby/Unknown/NotApplicable,
with two-strike flap damping — a failing observation only replaces a good state
after 2 consecutive failures, while recovery is immediate (damping suppresses
false alarms; symmetric damping would just make them linger).

53 tests green, 0 warnings in Debug and Release. Tests for the validator, the
client golden payloads (with AND without `data`, so a partly-bumped fleet renders)
and the derivation table are front-loaded here from task 3.8 so this batch is
verified rather than pending.

Also pins AngleSharp 1.5.2 test-side: bunit 1.40.0 pulls 1.2.0, which trips NU1902
and therefore the warnings-as-errors gate (same fix HistorianGateway made in
6bc005d).
2026-07-24 06:42:21 -04:00
Joseph Doherty 80668a07bd feat(health): 0.2.0 — optional per-entry data + Akka cluster-view
Phase 0 of docs/plans/2026-07-22-overview-dashboard-impl-plan.md: give the
canonical health JSON a structured channel so the family overview dashboard can
read each Akka cluster's current leader.

- ZbHealthWriter: optional `"data": {...}` per entry, sourced from
  HealthReportEntry.Data, emitted only when non-empty. Per-property JsonIgnore
  (NOT a global DefaultIgnoreCondition) so `"description": null` still renders —
  payloads from data-less checks stay byte-identical to 0.1.0.
- AkkaClusterHealthCheck: BuildClusterData publishes this node's own view —
  leader (omitted while unknown), selfAddress, selfRoles (sorted), memberCount,
  unreachableCount — on every result path. The startup-safety paths (no
  ActorSystem / cluster inaccessible) stay description-only.
- Tests: writer data emit/omit (raw-JSON assert on the omit case), and a real
  single-node self-joined cluster via Akka.TestKit.Xunit2 for the data values.
  70 tests green (25/39/6).
- Version 0.1.0 -> 0.2.0; 3 packages published to the Gitea feed and
  restore-verified from a scratch consumer, which serves data.leader live.
2026-07-24 05:38:36 -04:00
Joseph Doherty 6721c8a224 docs(index): OtOpcUa Phase 6 (mesh partition) done+merged — per-cluster meshes, per-Cluster Primaries
Per-cluster mesh Phase 6 merged to OtOpcUa master 2839deb5 (pushed 2026-07-24).
Records the split into one 2-node mesh per application Cluster, the consumer-visible
per-Cluster ServiceLevel/Primary scoping (two+ Primaries fleet-wide by design, no
wire-contract change), the SplitTopologyTransportValidator, pair-local secrets, and
the deferred live gate.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 04:40:23 -04:00
Joseph Doherty 3dbd6cd57c docs(index): OtOpcUa per-cluster mesh Phase 5 done (gRPC telemetry stream, merge 35552a96)
Phases 0a/0b/1/2/3/4/5 done; tip 35552a96. Phase 5 = gRPC live-telemetry stream
behind Telemetry:Mode/TelemetryDial:Mode dark switch; not consumer-facing (central<->node
observability, no OPC UA wire-contract change).

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-23 17:22:18 -04:00
Joseph Doherty 92015da4e0 docs: ScadaBridge OtOpcUa v3 nsu= cutover phase 2 shipped, A/B/C deferred (ScadaBridge #14) 2026-07-23 15:23:35 -04:00
Joseph Doherty 1911de2c3c docs(index): OtOpcUa per-cluster mesh Phase 4 done (cut driver ConfigDb)
Phase 4 merged + pushed (tip 3a590a0c): driver-only nodes run with no ConfigDb
connection string. Consumer-visible ServiceLevel change — a DB-less site node
now holds 240/250 with central SQL down (survive-alone), where a DB-backed node
would drop to 0/100. Phases 0a/0b/1/2/3/4 done; 5-7 not started.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-23 14:35:30 -04:00
Joseph Doherty b5e7bc8b3c docs(index): OtOpcUa per-cluster mesh Phases 1/2/3 done (config fetch-and-cache)
Update the OtOpcUa entry: the per-cluster mesh election-scoping fix now has Phases
0a/0b/1/2/3 merged + pushed (tip d01b0695). Phase 2 = ClusterClient command transport
(MeshTransport:Mode dark switch). Phase 3 = config fetch-and-cache — a FetchAndCache
driver node boots + serves last-known-good with central SQL down (extends the #485
availability guarantee consumers like ScadaBridge rely on); a failed fetch is 'no answer',
never an empty config. Live gate caught + fixed a Status==Sealed serve-gate deadlock.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-23 11:09:17 -04:00
Joseph Doherty 4d93d2580b docs(akka): ClusterClient→gRPC migration design + chunking addendum
akka_msquic_transport.md gains §8a (Akka.Cluster.Chunking / Akka.Delivery —
the 1.5-era in-cluster large-message tool, why it can't cross the ClusterClient
boundary) and the Artery-now-in-development §9 fix.
scadabridge_clusterclient_to_grpc.md: full migration assessment — ClusterClient
inventory (two-actor choke points, 29 commands incl. one dead), gRPC-only
design (topology, oneof contracts, active-node analysis: singleton-routed so
transport needs only either-node failover+failback), PSK-from-Secrets auth,
frame-cap-raise tradeoffs, and §7 deep-dive corrections. Companion impl plan
lives in ScadaBridge/docs/plans.
2026-07-22 17:10:30 -04:00
Joseph Doherty 6dd7858619 docs(overview): family overview dashboard — options, design, impl plan, mockup
Options investigation (Aspire standalone/HealthChecks.UI/Gatus/Homepage/
Grafana vs custom) → decision: custom ZB.MOM.WW.Overview Blazor app in the
family look. Design: anonymous read-only single pane, appsettings registry
of all four apps' instances, cache-first poller on a configurable timer,
configurable staleness timeout, Active/Standby from /health/active, and
per-cluster Akka leader chips (needs Health 0.2.0 optional per-entry data).
Impl plan: phased + code-verified — Phase 0 Health 0.2.0 (writer data field
+ AkkaClusterHealthCheck cluster view), Phase 1 ScadaBridge site-node
MapZbHealth PR (:8084; role-scoped active check), Phase 2 OtOpcUa bump,
Phase 3 the app (:5320), Phase 4 live rig acceptance. Mockup is the visual
reference: real Theme tokens + embedded IBM Plex, all card states incl.
stale + split-brain. Ready to execute in a fresh session.
2026-07-22 09:22:13 -04:00
Joseph Doherty ff58ad4f60 docs(akka-msquic): 1.6 status update — Artery-TCP implementation began 2026-07-03
Corrects the same-day report: PR searches for 'quic' missed work named
'artery'. Artery is landing rapidly on upstream dev as Artery-TCP (openspec
#8313; framing/handshake G1-G2; parallel lanes #8356/#8357; large-message
streamId 3 #8352; MNTR/CI for both transports), with Serialization.V2 in
parallel. TCP-first resolves §7's forward note: pipeline benefits arrive
without libmsquic, dropping the §6 platform constraints from the 1.6
adoption path. No previews or public benchmarks yet.
2026-07-22 06:49:39 -04:00
Joseph Doherty dbfd5e7a7e docs: Akka.NET transport deep-dives — MsQuic/1.6 Artery and multi-DC clusters
Two research reports (2026-07-22):
- akka_msquic_transport.md: what a QUIC transport on Akka 1.5.62 would take,
  the 1.5-vs-1.6 pipeline delta that gates the benefits, .NET-native
  alternatives (managed sockets ranked first), and measured throughput
  evidence (PR #4594: transport swap is throughput-neutral).
- akka_multidc_clusters.md: JVM multi-DC semantics mapped onto the
  central-pair + site-pairs topology, the stalled Akka.NET port (#3261 /
  PR #4111), per-system fit vs the OtOpcUa mesh program, and the
  before-Phase-2 decision point.
Recommendation in both: don't build; track upstream.
2026-07-22 06:41:36 -04:00
Joseph Doherty 5919e29969 docs(index): record OtOpcUa's two redundancy corrections and the NU1903 pin
Both are consumer-visible, so they belong in the index rather than only in the
repo. The downing strategy moved to AutoDowning because a two-node OtOpcUa pair
on keep-oldest could not survive a crash of the oldest node — the same defect and
the same fix as ScadaBridge cf3bd52f, which is where the analysis came from. And
the redundancy Primary is now the oldest Up driver member rather than the role
leader, which changes which node advertises ServiceLevel 250 after a restart —
relevant to anything selecting a server by ServiceLevel.

Also notes that the four NU1903 advisories which broke ScadaBridge's fresh
restores were breaking OtOpcUa's too, and that the election remains scoped per
Akka cluster until the per-cluster mesh work lands.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-21 19:05:27 -04:00
Joseph Doherty ede5275d3a docs(index): OtOpcUa address-space availability guarantee; track the OtOpcUa LocalDb design
The #485/#486 fixes changed nothing structural about OtOpcUa — same drivers,
same dual-namespace scheme, same Galaxy flow — so the row is mostly untouched.
What DID change is a fact any OPC UA consumer depends on, and ScadaBridge is
mid-cutover onto exactly this address space: a transient ConfigDb error used to
empty the served address space (a PureRemove of every node, observed live and it
stayed empty), stop every driver and clear every subscription, while still
reporting the deploy Applied. A consumer would have seen all bindings go bad
with nothing upstream admitting a problem. Recorded on the OtOpcUa row.

Also fixes real drift found while checking: the OtOpcUa LocalDb adoption design
was sitting UNTRACKED here and unreferenced by the index, even though the
LocalDb row claims the adoption designs live in this repo. Now committed,
referenced alongside the ScadaBridge one, and its stale header corrected —
it still said Phase 1 was "not yet executed" against lib 0.1.1, when Phase 1
shipped and merged against 0.1.3. Phase 2 designed but not executed, on both
the doc and the row.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-21 03:30:36 -04:00
Joseph Doherty 4f33eccfa7 Merge fix/localdb-wiped-peer-snapshot: LocalDb 0.1.3 rebuilt-peer replication fixes
Two defects about a node whose LocalDb file is lost, the second only reachable
once the first was fixed:

- 0.1.2 — a converged pair prunes its oplog to empty on ack, and snapshot
  detection read an empty oplog as 'no gap possible'. The steady state of a
  healthy pair was the one state from which a rebuilt node could never be
  back-filled.
- 0.1.3 — with back-fill working, the rebuilt node's OWN writes were silently
  dropped: last_applied_remote_seq is a watermark in the peer's seq space and a
  rebuilt peer numbers from 1, so the healthy node's stale watermark made the
  sending side skip its whole oplog.

Both found by the OtOpcUa LocalDb Phase 1 live gate on the docker-dev rig and
reproduced offline as RED tests first. 149/149 pass.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-21 02:17:42 -04:00
Joseph Doherty 7c8dd654ba docs(index): LocalDb 0.1.3 — the rebuilt-peer write-drop fix
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-21 02:13:48 -04:00
Joseph Doherty 3b7489a7b0 fix(localdb): a rebuilt peer's own writes were silently dropped
The other half of the rebuilt-peer story, and only reachable once 0.1.2 made
back-fill work: with the wiped node repopulated and writing again, its writes
never reached its peer.

last_applied_remote_seq is a watermark in the PEER's seq space, and a rebuilt
peer is a new database whose oplog numbers from 1. Two things then went wrong,
and both had to be fixed — the first alone leaves data stuck:

- The healthy node kept advertising the OLD peer's watermark, and the sending
  side seeds its pump from exactly that number, so the rebuilt node skipped its
  entire oplog. The watermark (and the observed peer clock) is now reset when
  the peer's node id changes. last_acked_seq is deliberately NOT reset: it
  describes our own oplog's pruned horizon and is what tells a rebuilt peer it
  needs a snapshot — clearing it would turn 0.1.2's back-fill back off.

- A peer claiming to have applied more of our stream than we have ever produced
  can only be remembering a previous incarnation of us. That claim is now
  rejected in favour of starting from the beginning; LWW makes the re-send
  harmless. This mirrors the clamp RecordPeerAckAsync already applies to acks.

Found on the OtOpcUa docker-dev rig, where the healthy node held watermark 10
while the rebuilt peer's oplog was seqs 1-3 and its last_acked stayed 0 — the
peer never accepted a single one. Covered by a test that converges with one
peer, replaces it with a fresh database, and requires a write on the new peer
to arrive: RED before this fix, green after. 149/149 pass.

Version 0.1.3.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-21 02:06:03 -04:00
Joseph Doherty 45d08929e7 docs(index): LocalDb 0.1.2 + OtOpcUa Phase 1 adoption in the umbrella index
The lib row still claimed ScadaBridge was the only consumer (verified
2026-07-20); OtOpcUa's Phase 1 adoption merged 2026-07-21. Records 0.1.2 and
the wiped-peer snapshot fix it carries.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-21 01:24:09 -04:00
Joseph Doherty cad3bcb6bf fix(localdb): snapshot a wiped peer whose deltas were already acked and pruned
A converged pair prunes its oplog on ack, so "no oplog rows at all" is the
steady state of a healthy pair — not an anomaly. ComputeSnapshotRequiredAsync
measured the peer's gap against the oldest surviving oplog row and treated an
empty oplog as "no gap possible", which made that steady state the one state
from which a wiped peer could never be healed: it reconnected with an empty
database and a zero watermark, was told it needed no snapshot, and stayed
empty indefinitely.

With an empty oplog the oldest seq we could still stream is one past
last_acked_seq — ack-pruning is the only thing that empties the oplog without
also flagging needs_snapshot, and it deletes exactly seq <= last_acked_seq.
Measuring the gap against that heals the wiped peer and leaves every other
case identical: a converged peer's watermark equals last_acked_seq (no
snapshot), and a pair that has never written anything has both at zero.

Found by the OtOpcUa LocalDb Phase 1 live gate (check 4), which recorded it as
a documented limitation. Covered by a test that reproduces the live-gate
scenario through the real session stack: RED before this fix (the wiped side
never converges, 15 s timeout), green after. 148/148 pass.

Version 0.1.2.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-21 01:17:32 -04:00
Joseph Doherty c42348eff5 docs(localdb): the ScadaBridge adoption is merged, not pending
Phases 1 and 2 landed on ScadaBridge main together as PR #23
(merge 28ca04d7) on 2026-07-20 and were pushed to origin. The
component row still described both as unmerged branches.

Verified before editing: main..feat/localdb-phase1 and
main..feat/localdb-phase2 are both empty, main == origin/main,
and SiteReplicationActor/ReplicationService have no hits left in
main's tree.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-20 08:41:21 -04:00
Joseph Doherty 5836c79d3d docs(localdb): phase 2 complete + 0.1.1 in the component row
The row still said "Adopted by ScadaBridge (Phase 1 only)" and "Phase 2 is NOT
started". Both phases are now complete with a 10-check live gate PASS.

Also records what the row could not have said before: the 0.1.1 parent-
directory boot fix, that ScadaBridge is the library's ONLY consumer (verified —
OtOpcUa, mxgw and HistorianGateway have no LocalDb reference at all), the
deliberate never-registered status of notification_lists/smtp_configurations
and why, and the three operational constraints an operator has to know
(stop/start a pair together, the TombstoneRetention resurrection bound, and
MaxBatchSize batching by row count against a 4 MB gRPC cap).

Replication remains default-OFF and no production deployment has it enabled.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-20 05:38:02 -04:00
Joseph Doherty 10d076a9da test(localdb): sustained concurrent write load through pooled connections
Closes criterion 5 of ScadaBridge/docs/known-issues/2026-07-20-localdb-disk-io-
error-under-load.md §9: Phase 1 shipped only correctness and convergence tests,
so nothing exercised many pooled per-operation connections writing a REGISTERED
table (capture trigger + zb_hlc_next UDF) concurrently — which is exactly the
shape the ScadaBridge site nodes run under load.

8 writers x 250 inserts, one fresh pooled connection per operation, with
concurrent readers churning the pool and the WAL index throughout. Asserts both
that every write lands and that the oplog captured each exactly once, so a
trigger that dropped or double-counted under contention fails it.

Written during the 2026-07-20 disk-I/O incident investigation and left
uncommitted; the incident itself was root-caused to host-side sqlite3 access on
a bind-mounted WAL database, not a library defect.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-20 05:01:57 -04:00
Joseph Doherty 12e6f0d2c5 fix(localdb): create the database file's parent directory (0.1.1)
SQLite creates the database FILE on demand but never its parent DIRECTORY, and
SqliteLocalDb opens the file eagerly in its constructor — so a configured path
under a not-yet-existing directory was a hard startup failure ("SQLite Error
14: unable to open database file") rather than a first run that just worked.
Relative defaults like ./data/app.db hit it routinely; a containerized consumer
escapes only when a volume mount happens to create the directory first.

ScadaBridge found this during LocalDb Phase 2 and worked around it app-side
(SiteLocalDbDirectory.Ensure). Every other consumer still had the gap, so the
guarantee belongs here.

The standing objection to a library that creates directories is that it can
mask a mis-typed path by silently starting an empty database. That does not
apply: SQLite ALREADY does exactly that for a mis-typed file NAME. Refusing to
create the directory protects nothing — it only makes the two halves of one
path behave inconsistently and turns the directory half into an opaque error.

A creation failure (permissions, read-only mount) now throws an
InvalidOperationException naming the directory and the cause, instead of
surfacing as SQLite error 14. That is a change of exception type, but only on a
path that already failed hard.

Both behaviours are pinned and verified to fail with the call removed.
147 tests pass, 0 warnings. Published to the Gitea feed at 0.1.1.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-20 04:51:18 -04:00
Joseph Doherty baea6cc2e0 docs(secrets): interactive-console quickstart + discoverability sweep
- new task-first quickstart (deployment seeding + KEK-lockout walkthroughs)
- runbook: document the sixth ReferenceStatus (PresentUnverified) + quickstart link
- README + shared-contract + umbrella CLAUDE.md secrets row updated for 0.3.0 console
- CLI usage text now mentions the interactive console (was undiscoverable from --help)
2026-07-19 14:55:53 -04:00
Joseph Doherty 0139c90f89 docs(localdb): correct the adoption-design doc location in the LocalDb row
The design doc lives here in scadaproj, not in ScadaBridge; ScadaBridge holds
the Phase 1 plan and the Phase 2 gate.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-19 11:27:53 -04:00
Joseph Doherty 87ffe1b774 Merge branch 'worktree-secrets-interactive-cli' 2026-07-19 11:27:50 -04:00
Joseph Doherty 980b4a37f6 fix(secrets-cli): contain file-source key IO/access errors in doctor + bundle-import flows 2026-07-19 10:38:16 -04:00
Joseph Doherty 6dbb372adf docs(secrets): interactive console runbook + 0.3.0 version bump 2026-07-19 10:37:37 -04:00
Joseph Doherty 1f7d26d4d9 feat(secrets-cli): launch interactive console on bare secret in a TTY 2026-07-19 10:30:30 -04:00
Joseph Doherty 7e67bd61c7 fix(secrets-cli): doctor flow — contained old-key errors, corrupt-row + rewrap-failure coverage 2026-07-19 10:27:54 -04:00
Joseph Doherty 1b65e820fb test(secrets): serialize env-var-mutating test classes into one xunit collection
Environment.SetEnvironmentVariable/GetEnvironmentVariable mutate the
process-wide environ; concurrent setenv/getenv across parallel xunit test
classes is a documented crash/flake source on glibc even with unique
variable names. Group the six classes that touch process env vars into a
DisableParallelization=true collection so they run serially against each
other while everything else keeps running in parallel.
2026-07-19 10:27:09 -04:00
Joseph Doherty 04018aab6e fix(secrets-cli): bundle import — source-key verification, format gate up front, access-denied containment 2026-07-19 10:00:59 -04:00
Joseph Doherty 4549b93d2a fix(secrets-cli): doctor flow — catch scoped exactly to the rewrap call 2026-07-19 10:00:00 -04:00
Joseph Doherty e8b34be255 fix(secrets-cli): audit flow — cipher guard, shared actor helper, summary + branch coverage 2026-07-19 09:55:16 -04:00
Joseph Doherty d1ec0fe5da feat(secrets-cli): lockout-recovery flow — KEK diagnosis, guided rewrap, lost-KEK reset 2026-07-19 09:53:10 -04:00
Joseph Doherty 000d283c99 feat(secrets-cli): bundle export/import flows 2026-07-19 09:52:31 -04:00
Joseph Doherty 08c4ce5769 docs(localdb): ScadaBridge Phase 1 adopted - LocalDb component row truth pass
Was 'no app adoption yet'. Now records the honest state: ScadaBridge site nodes
adopted Phase 1 on an UNMERGED branch, replication wired but default-OFF and
live-proven on the docker rig's site-a pair only, Phase 2 not started, no other
app adopted, no production deployment replicating.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-19 09:51:55 -04:00
Joseph Doherty e5a4d0276a fix(secrets-cli): CRUD flows — escaped validation markup, shared prompt helpers, delete existence check 2026-07-19 09:47:10 -04:00
Joseph Doherty 6ad710bdbd test(secrets-cli): structural no-plaintext assert (base64 '+' JSON-escaping flake) + override doc note 2026-07-19 09:46:26 -04:00
Joseph Doherty 779f8a8d78 feat(secrets-cli): deployment seeding flow — audit ${secret:} refs and fill the gaps 2026-07-19 09:43:17 -04:00
Joseph Doherty fe3ea684f5 feat(secrets-cli): CRUD flows for the interactive console 2026-07-19 09:37:46 -04:00
Joseph Doherty 138e2c1006 fix(secrets-cli): bundle import — verbatim ApplyReplicated writes, format gate, entry-aware errors 2026-07-19 09:35:39 -04:00
Joseph Doherty 4576f73c4c fix(secrets-cli): KEK doctor — fresh-row verdicts, degraded rewrap guard test, vanished-row visibility 2026-07-19 09:33:53 -04:00
Joseph Doherty e60f2d5dad fix(secrets-cli): reference auditor — InvalidName containment, comment-key parity, name normalization 2026-07-19 09:27:13 -04:00