312216ff2b9f96d99710dc9caa9ff143b91a1297
9 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Joseph Doherty
|
2e4e41a8f7 |
fix(auditlog): site audit DB onto the data volume; required path + soft flush
Closes WP1.2 of the arch-review remediation plan (finding #2, High): SqliteAuditWriterOptions.DatabasePath defaulted to CWD-relative "auditlog.db", which on the docker rig resolves onto the container's ephemeral overlayfs (not the mounted /app/data volume), silently discarding the pending audit forward-state backlog on every recreate; nothing in docker/ or docker-env2/ overrode it; FlushIntervalMs was validated but never read by the writer loop (one commit per event even at trickle rate); and no PRAGMA synchronous was set (SQLite's FULL default fsyncs every commit). - DatabasePath now has no default (mirrors ZB.MOM.WW.LocalDb's LocalDbOptions.Path) and is required pre-host for Site nodes only, via a new StartupValidator raw-config check (top-level "AuditLog:SiteWriter:DatabasePath", NOT nested under ScadaBridge: AddAuditLog binds that section off the configuration root). SqliteAuditWriterOptionsValidator deliberately does NOT check DatabasePath itself, because AddAuditLog runs its ValidateOnStart on both Central and Site composition roots but only Site nodes ever resolve the writer — checking it there would fail Central's boot too. - All 8 site-node appsettings under docker/ and docker-env2/ now set AuditLog:SiteWriter:DatabasePath to /app/data/auditlog.db (mounted volume, survives container recreate, same convention as LocalDb:Path); the local-dev base appsettings.Site.json sets ./data/auditlog.db to match. - The writer loop now honors FlushIntervalMs: after draining the immediately available burst, it keeps the transaction open (bounded by FlushIntervalMs from the first event) waiting for more trickle-rate events before committing, instead of flushing (and fsyncing) per event. - PRAGMA synchronous = NORMAL on the write connection — audit is best-effort by design (CLAUDE.md: "Audit-write failure NEVER aborts the user-facing action"), so NORMAL's narrower power-loss window is an acceptable trade for far fewer fsyncs; WAL mode still guarantees no corruption. - Tests: StartupValidator site-required/blank/central-exempt cases; writer trickle-load single-transaction coalescing + beyond-interval separate-transaction regression (new FlushCountForTests seam); options-validator doc updates reflecting the moved responsibility. Full suite runs green: AuditLog.Tests 368/368, Host.Tests 480/480. One-time migration note: the existing container-local auditlog.db (wherever it landed under CWD) is abandoned by this change, not migrated — already-forwarded rows are safe centrally (AuditLog is the durable copy), and any still-Pending rows on the abandoned path are lost once. This is the exact bug being fixed, not a new loss: those rows were already living outside the mounted volume and would not have survived the next container recreate regardless. Cross-reference docs/known-issues/2026-07-20-cached-telemetry-drain-hot-loop.md, which this placement bug caused. |
||
|
Joseph Doherty
|
7fd5cb2b56 |
feat(comm): Phase 4 — delete Akka ClusterClient site↔central transport, gRPC-only
ClusterClient→gRPC migration Phase 4 (docs/plans/2026-07-22-clusterclient-to-grpc-plan.md). Phases 2/3 proved both directions on gRPC; this removes the Akka transport underneath. Deleted: - AkkaCentralTransport, AkkaSiteTransport (+ their dedicated tests) - ISiteClientFactory + DefaultSiteClientFactory; CentralCommunicationActor legacy ctor + SelectTransport (Host now builds GrpcSiteTransport and injects it) - ClusterClient creation + both ClusterClientReceptionist.RegisterService calls in AkkaHostedService; the RegisterCentralClient message + receive block - CommunicationOptions.CentralContactPoints; the CentralTransport/SiteTransport coexistence flags; the CentralTransportMode/SiteTransportKind enums gRPC is now the only site↔central transport (site→central CentralControlService via GrpcCentralTransport; central→site SiteCommandService via GrpcSiteTransport), both built unconditionally by the Host. NoOpCentralTransport is the fail-loud null-default so TestKit command-dispatch suites still construct the site actor without a wired transport; production always injects GrpcCentralTransport. Config: CentralGrpcEndpoints is now unconditional — CommunicationOptionsValidator rejects blank entries (role-agnostic), and StartupValidator requires a Site node to list >=1 endpoint (fail-fast, mirrors GrpcPsk). Rig configs moved CentralContactPoints -> CentralGrpcEndpoints (docker x6, docker-env2 x2, Host default, deploy/wonder-app-vd03). Kept Akka.Cluster.Tools (ClusterSingleton still used). Tests: build 0/0; Communication.Tests 640, Host.Tests 421 green. Removed the ClusterClient.Send per-site-routing tests (covered by the transport suites), swapped the ISiteClientFactory-based ctors to a substitute ISiteCommandTransport, converted the audit-push integration relay to an in-process bridge transport. Docs: Component-Communication/Host/StoreAndForward, components/Communication, topology-guide, grpc_streams (SUPERSEDED note), the frame-size known-issue (retired amendment), and CLAUDE.md transport decisions. Not included: the dead IntegrationCallRequest path (#32) is a separate user-owned behavioral decision — SiteEnvelope routing is transport-agnostic so it still compiles. |
||
|
Joseph Doherty
|
2ee84af1c0 |
feat(grpc): PSK-authenticate the site gRPC control plane; drop the vestigial management receptionist registration
Phase 0 of the ClusterClient→gRPC migration
(docs/plans/2026-07-22-clusterclient-to-grpc-plan.md). Standalone hardening: it
closes a gap that exists today and is a precondition for moving command/control
onto gRPC in later phases.
T0.1 — delete the ManagementActor ClusterClientReceptionist registration.
It was built for an out-of-cluster CLI that was never written: the shipped CLI
speaks HTTP Basic to /management, which asks the actor in-process through
ManagementActorHolder. Nothing in the repo ever sent to /user/management. The
actor still runs there; only the cross-boundary advertisement is gone. Six
documents claimed the CLI used ClusterClient — including the CLI's own README
"Architecture Notes" — and are corrected here rather than left to rot.
T0.2 — record, do not port, the dead integration-routing path.
IntegrationCallRequest is unwired at BOTH ends: RouteIntegrationCallAsync has
zero callers anywhere, and RegisterLocalHandler(Integration, …) appears only in
a test, so production always answers "Integration handler not available". It is
excluded from the gRPC contract (28 of 29 commands migrate) rather than
enshrined on an additive-only wire format, and deleting it during a
transport migration would mix a behavioural change into a change whose whole
value is that behaviour is identical. See
docs/known-issues/2026-07-22-integration-call-routing-is-dead-code.md.
T0.3 — preshared-key authentication on SiteStreamService.
The service shipped with no auth at all: plaintext h2c, no interceptor, so
anything that could reach a site node's :8083 could open a live data stream or
read audit rows back via PullAuditEvents/PullSiteCalls. ControlPlaneAuthInterceptor
now gates /sitestream.SiteStreamService/ — modeled on LocalDbSyncAuthInterceptor
(constant-time compare, fail-closed, PermissionDenied) but gating a SET of
service prefixes so phases 1A/1B add services rather than interceptors. LocalDb
sync keeps its own separate key: it authenticates the pair partner, not central,
and collapsing the two would make a site's central-facing key also admit writes
into its database.
Keys are per site (SB-GRPC-PSK-<siteId>), never fleet-wide, so a compromised
site yields only its own. Central attaches them through ControlPlaneCredentials,
which binds CallCredentials to the channel — covering unary and streaming
uniformly, and letting the key resolve asynchronously, which a client
interceptor could not do without blocking. All three central→site channel
creation sites go through it (SiteStreamGrpcClient and both audit pull invokers);
the pull invokers' channel caches are re-keyed by (site, endpoint) because
credentials are per-site and bound to the channel.
Two decisions beyond the plan:
* StartupValidator now requires GrpcPsk on Site nodes. The plan specified only
the runtime gate, but fail-closed with no boot check produces a node that
joins, answers heartbeats and reports healthy while refusing every stream,
audit pull and telemetry ingest — silent and total. Same reasoning as the
existing inbound API-key pepper rule.
* Added Communication:SitePsks as a central-side key map. The plan assumed
central would read the store, seeded via a dev KEK; the docker rig
deliberately boots with no master key, so store-only resolution would leave
it unable to dial its own sites. The store stays primary — it is the only
source that can serve a site added at runtime — with the map covering
key-less hosts and one-off pins. Neither source falling back to
"unauthenticated" is the invariant.
T0.4 — dev keys on both rigs and tests.
34 tests. The seven that matter most exercise a real in-process gRPC stack over
TestServer: the unit tests on either side of the wire would both stay green if
the halves disagreed, and gRPC refuses call credentials on a plaintext channel
by default — the UnsafeUseInsecureChannelCallCredentials opt-in is only provable
by making a real call. They confirm correct key passes on unary AND streaming,
wrong key and no-credentials both get PermissionDenied, and an unresolvable key
fails the call with nothing reaching the service.
OPERATIONAL: a site node upgraded to this build without a key will not boot.
That includes the gitignored deploy/wonder-app-vd03/ overlay.
|
||
|
Joseph Doherty
|
cf3bd52f93 |
feat(cluster): auto-down downing strategy — either-node crash now fails over (owner decision 2026-07-21: availability over partition-safety)
Two-node keep-oldest could NEVER survive a crash of the oldest/active node:
Akka.NET 1.5.62 KeepOldest.OldestDecision only lets down-if-alone rescue a
side with >= 2 members, so the 1-vs-1 survivor takes DownReachable and downs
ITSELF — proven live on the rig ('SBR took decision ... including myself')
before this change. static-quorum(1) is worse (IsTooManyMembers -> DownAll);
keep-majority just re-keys the fatal crash to the lowest address.
SplitBrainResolverStrategy gains 'auto-down' (new default): BuildHocon emits
Akka's AutoDowning provider with auto-down-unreachable-after = StableAfter.
The leader among the REACHABLE members downs the unreachable peer, so the
survivor takes over singletons and /health/active in ~25s regardless of which
node died. Accepted trade (explicit owner decision): a real network partition
runs dual-active until an operator restarts one side. keep-oldest remains
supported; DownIfAlone validation is now scoped to it.
Live drill on the rebuilt rig: active-crash TAKEOVER in 28s (victim still
down; all 7 singletons Younger->Oldest), standby-crash removal 27s with 0
routing blips; victims rejoin as standby in 2s. New real-cluster tests pin
both directions (SbrFailoverTests.AutoDown_*); TwoNodeClusterFixture gains a
strategy knob. All 16 appsettings flipped (src, docker, docker-env2, and the
gitignored wonder-app-vd03 overlay on disk — owner must sync to the host).
Docs: decision record docs/plans/2026-07-21-auto-down-availability-decision.md,
Component-ClusterInfrastructure downing section rewritten, drill + README
reworked (active mode now asserts takeover), deferred-work SBR row resolved.
|
||
|
Joseph Doherty
|
605e56829e |
chore(config): retire ReplicationEnabled, make legacy db paths migration-only
LocalDb Phase 2 deleted the bespoke replicators, so three config keys changed meaning or died outright: - ScadaBridge:StoreAndForward:ReplicationEnabled is fully dead. Deleted the property, its 10 config entries, and the 5 test references. - SqliteDbPath / SiteDbPath are now migration-only: they name the legacy files SiteLocalDbLegacyMigrator drains at boot, not live databases. Both mandatory rules are relaxed accordingly (StartupValidator's Site-only Require, and the S&F validator's non-empty rule) — an absent value now means "nothing to migrate", so an already-migrated node can drop the key. DatabaseOptions- Validator still rejects a present-but-blank value. - SiteRuntime:ConfigFetchRetryCount's only reader was SiteReplicationActor. Deleted with its validator rule. The two path keys stay present in every config, now with a comment explaining why: removing them would strand un-migrated data on a node that has not yet started once. Both relaxations are pinned by the inverse of the test they replace (Site_MissingSiteDbPath_IsAccepted..., EmptySqliteDbPath_IsAccepted...), each verified to fail with the old rule restored. Note: deploy/wonder-app-vd03/appsettings.Site.json is under a gitignored deploy/ tree, so its edit is local-only and must be repeated on the box. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts |
||
|
Joseph Doherty
|
2977e6c45c |
feat(localdb): register the consolidated site database in the composition root
Task 3. Site nodes now open one ZB.MOM.WW.LocalDb-managed database holding OperationTracking and site_events as replicated tables. SiteLocalDbSetup.OnReady applies both schemas and then RegisterReplicated's them. That order is load-bearing and easy to get silently wrong: capture is trigger-based CDC, so rows written before registration are never captured or snapshotted - they would be invisible to the peer forever, with no error anywhere. Storage ships UNCONDITIONALLY, no feature flag. With no peer configured LocalDb is just a local SQLite file, so this is behaviour-equivalent to what site nodes do today. Replication is opt-in and lands in Program.cs (Task 7), which owns the gRPC host the sync endpoint needs. Config: LocalDb:Path added to all EIGHT site-node appsettings (the plan said six - docker-env2's site-x pair exists too), plus the dev template and the wonder-app-vd03 production sample. All ten are required, not optional: LocalDbOptions.Path is validated with ValidateOnStart, so a site node without it fails fast at startup. That is the desired posture, and it is what caught ActorPathTests - the one Host fixture that actually starts a host - whose config now sets the path too. Note the path choice fixes an existing data-loss bug in passing: site-tracking.db and site_events.db both defaulted to CWD-relative paths (/app/...), outside the mounted volume, so they were silently lost on every container recreate. The consolidated database lives on /app/data. Tests are container-built against the REAL SiteServiceRegistration.Configure, not a hand-assembled ServiceCollection - this family has shipped three wiring defects that only a real graph would catch (inert Secrets replicator 0.2.0, singleton deadlock 0.2.2, Secrets.Ui missing IAuditWriter). They assert the library's own view of what is registered rather than that we called the right method. Verified (all after the change): dotnet build ZB.MOM.WW.ScadaBridge.slnx -> 0 Error(s), 0 Warning(s) Host.Tests -> 294 passed (5 new, red-first) SiteRuntime.Tests -> 530 passed CentralUI.Tests -> 925 passed Commons.Tests -> 684 passed Communication.Tests -> 312 passed SiteEventLogging.Tests-> 70 passed Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts |
||
|
Joseph Doherty
|
7c1d61647e | feat(dcl): bound per-tag seed ReadAsync with SeedReadTimeout (#232, DCL-027) | ||
|
Joseph Doherty
|
7b0b9c7365 |
refactor: rename ScadaLink → ZB.MOM.WW.ScadaBridge (code + projects + namespaces)
Solution + 23 src projects + 26 test projects renamed; folders, csproj, namespaces, and ScadaLinkDbContext/ScadaBridgeDbContext class updated. ActorSystem "scadalink" → "scadabridge", Akka seed-node URLs migrated. SQL roles/logins, LDAP domains, CLI command name, and CLI config dir (~/.scadalink → ~/.scadabridge) also renamed. Build green; 5 Host.Tests fail awaiting SQL login rename in next commit. Pre-existing StaleTagMonitor timing flakes unchanged. Rename script committed at tools/rename-to-scadabridge.sh. |
||
|
Joseph Doherty
|
c252a80f9d | feat(docker-env2): add site-x appsettings |