dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'fix/standby-secret-mirror-status' (M1: standby mirror failures on the health report)

Browse Source 
# Conflicts:
#	CLAUDE.md
#	docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md
This commit is contained in:
Joseph Doherty
2026-09-28 21:37:03 -04:00
parent 253939cd5d f5756affef
commit 4e6409184c
29 changed files with 1738 additions and 106 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+5 -5
View File
@@ -90,7 +90,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
- DCL write failures returned synchronously to calling script.
- Tag path resolution retried periodically for devices still booting.
- Static attribute writes persisted to local SQLite (survive restart/failover, reset on redeployment).
- **Consolidated site database (LocalDb Phase 1 + 2, complete 2026-07-20).** Eleven tables now live in ONE `ZB.MOM.WW.LocalDb`-managed SQLite file (the `SiteLocalDbSetup.ReplicatedTables` list — the 11th, `deployed_secrets`, was added 2026-09-28 by the site-secrets-via-artifact-deploy change, see *Templates & Deployment* below), configured by the **required** `LocalDb:Path` (`/app/data/site-localdb.db` on the rig; validated with `ValidateOnStart`, so a site config missing it fails to boot). Consequences worth knowing:
- **Consolidated site database (LocalDb Phase 1 + 2, complete 2026-07-20).** Twelve tables now live in ONE `ZB.MOM.WW.LocalDb`-managed SQLite file (the `SiteLocalDbSetup.ReplicatedTables` list — the 11th, `deployed_secrets`, was added 2026-09-28 by the site-secrets-via-artifact-deploy change, see *Templates & Deployment* below; the 12th, `secret_mirror_status` — one row per node carrying that node's deployed-secrets mirror failures so the standby's reach central on the active node's health report — by review finding M1 the same day), configured by the **required** `LocalDb:Path` (`/app/data/site-localdb.db` on the rig; validated with `ValidateOnStart`, so a site config missing it fails to boot). Consequences worth knowing:
- `site_events.id` changed from autoincrement INTEGER to an application-minted **GUID**. Last-writer-wins keys on the primary key, so two nodes independently minting `id=1,2,3…` would destroy each other's events rather than merge them. The event-log read path uses a composite `(timestamp, id)` keyset cursor with an **opaque string** continuation token; `EventLogEntry.Id` and both `ContinuationToken`s are `string`/`string?` on the site↔central Akka DTOs.
- `ScadaBridge:OperationTracking:ConnectionString`, `ScadaBridge:SiteEventLog:DatabasePath` and — as of Phase 2 — `ScadaBridge:StoreAndForward:SqliteDbPath` + `ScadaBridge:Database:SiteDbPath` are all **migration-only** — nothing reads them but `SiteLocalDbLegacyMigrator`, which copies a pre-Phase-1 file in once (deterministic `mig-{NodeName}-{legacyId}` event ids, `INSERT OR IGNORE`, runs AFTER `RegisterReplicated` so migrated rows replicate) and renames it `.migrated`. Delete the keys once a node has migrated.
- This incidentally fixes a data-loss bug: both legacy databases defaulted to CWD-relative paths **outside** the mounted volume and were discarded on every container recreate.
@@ -101,10 +101,10 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
- **`notification_lists` and `smtp_configurations` are created but deliberately NOT registered.** They are permanently empty on a site (no writer since 2026-07-10, the migrator skips them, the active-node purge keeps them empty), and registering them would open a standing replication channel whose only historical payload was plaintext SMTP passwords. Pinned by a security-named test, and verified live: those two tables have **no CDC triggers** on either rig node.
- **Operational constraints (read before upgrading a site pair):** stop and start both nodes TOGETHER — rolling one at a time is no longer supported, since the legacy `SfBufferSnapshot` compatibility handler went with the replicator. And a node offline longer than `LocalDb:Replication:TombstoneRetention` (default 7 days) can resurrect deleted rows on rejoin. See `docs/deployment/topology-guide.md`.
- **Batching is by BYTE BUDGET as of LocalDb 0.2.0** — `LocalDb:Replication:MaxBatchBytes` (default **2 MB**, sized under the 4 MB gRPC cap) bounds a delta/snapshot message by summed serialized size via a per-message split in `SyncSession.PumpLoopAsync`, with `MaxBatchSize` demoted to a secondary row cap; a single row over budget is sent alone rather than stalling the stream. **The rig's old `MaxBatchSize = 16` pin (a hand-computed byte-budget proxy: ~70 KB worst-case `config_json` x the 500 default is ~35 MB) is retired, but `MaxBatchSize` is NOT fully redundant with `MaxBatchBytes`** — it also bounds the separate DB READ page in `OplogStore.ReadBatchAboveAsync`/`SnapshotStreamer`, which materializes the whole page into memory *before* the byte-budget split runs, so an unset (500-default) `MaxBatchSize` still lets a reconnect drain transiently allocate ~35 MB per read even though every wire message stays under `MaxBatchBytes` (arch-review adversarial finding F2). Both site-a nodes on `docker/` therefore pin an explicit `"MaxBatchSize": 64` to bound that transient allocation, while `MaxBatchBytes` stays unset (its 2 MB default) to bound the wire message; site-b/site-c stay unreplicated so the key doesn't apply there.
- **CDC registration is conditional, and both directions self-heal at boot** (arch-review WP1.3 + WP3.3, `Host/SiteLocalDbSetup.cs`). Capture triggers are installed only when this node has replication configured — `PeerAddress` **or** `ApiKey`, an OR because only the dialling half sets `PeerAddress` while the passive half carries the key alone. An unreplicated node calls **`DeregisterReplicated`** on all eleven tables at boot, dropping triggers an earlier build left behind and pruning their oplog/row-version rows (idempotent; logs once at Information when something was actually cleaned). A replicating node registers with **`baselineExistingRows: true`**, which seeds `__localdb_row_version` for pre-existing rows at the LWW floor (HLC `0`, this node's id) and flags a snapshot resync — so turning replication ON for a site that has been running without it now converges on the rows already in the file instead of only on writes made after the restart. Deregistration must be symmetric (the handshake compares registered-table digests fail-closed), so replication is a both-nodes-together change in either direction. LocalDb 0.2.0 also makes backlog depth O(1) and drops the unused `__localdb_oplog_hlc` index; the on-disk bookkeeping schema goes to v2, upgraded in place on open, with no wire change (0.1.x peers still sync).
- **CDC registration is conditional, and both directions self-heal at boot** (arch-review WP1.3 + WP3.3, `Host/SiteLocalDbSetup.cs`). Capture triggers are installed only when this node has replication configured — `PeerAddress` **or** `ApiKey`, an OR because only the dialling half sets `PeerAddress` while the passive half carries the key alone. An unreplicated node calls **`DeregisterReplicated`** on all twelve tables at boot, dropping triggers an earlier build left behind and pruning their oplog/row-version rows (idempotent; logs once at Information when something was actually cleaned). A replicating node registers with **`baselineExistingRows: true`**, which seeds `__localdb_row_version` for pre-existing rows at the LWW floor (HLC `0`, this node's id) and flags a snapshot resync — so turning replication ON for a site that has been running without it now converges on the rows already in the file instead of only on writes made after the restart. Deregistration must be symmetric (the handshake compares registered-table digests fail-closed), so replication is a both-nodes-together change in either direction. LocalDb 0.2.0 also makes backlog depth O(1) and drops the unused `__localdb_oplog_hlc` index; the on-disk bookkeeping schema goes to v2, upgraded in place on open, with no wire change (0.1.x peers still sync).
- **LocalDb 0.3.0 (2026-08-15) makes that snapshot flag PER-TABLE** — the residual 0.2.1 explicitly deferred. On-disk bookkeeping schema goes to **v3** (a new `__localdb_snapshot_state` table, upgraded in place on open); still wire-compatible, and `lib_schema_version` deliberately stays `1` because the handshake compares it fail-closed for *equality*. The extension is negotiated by **capability** (`Handshake.supports_partial_snapshot`, `SnapshotBegin.tables` where **empty == all tables**), so a peer that does not advertise it is served a full snapshot — a scoped one would make it advance `last_applied_remote_seq` past deltas it never received. For the same reason **a partial snapshot moves no watermark on either side**; the overlapping deltas are still pumped and absorbed by LWW. Snapshot debt now clears on the receiver's new `SnapshotAck` rather than on enqueue, so a crash mid-snapshot re-streams the same scope. **`SiteLocalDbSetup.cs` needed no restructuring** — it already calls `RegisterReplicated` once per table in a loop, which is exactly what per-table flagging keys off. The consequence to know: **adding an eleventh table to `ReplicatedTables` on an already-replicating site now snapshots that one table**, where through 0.2.x the same edit re-streamed all eleven in full in both directions. First-boot behaviour is unchanged (all ten seed at once ⇒ the flagged set is every registered table ⇒ an ordinary full snapshot), and so is upgrading the rig in place (the ten tables are already ledgered, nothing seeds, nothing is flagged). A future "register the whole list in one call" API would silently give the coarse behaviour back.
- **LocalDb 0.3.2 (2026-08-19, ScadaBridge arch-review P4-1 + P4-3; wire-compatible, no schema change):** (1) the sync pump now **faults the live session when a cap-prune invalidates its position** (horizon check on the oplog's min seq before each read batch + on the batch head after it; throws the new public `LocalDbOplogGapException` through the 0.3.1 fault-teardown) — previously a slow-but-alive peer silently skipped the pruned gap and the pair diverged until the next reconnect; (2) **transport keepalives both sides** — client `SocketsHttpHandler` pings via new validated `LocalDb:Replication:KeepAlivePingDelay`/`KeepAlivePingTimeout` (defaults 60s/20s, `Always` policy), and ScadaBridge's site gRPC listener sets Kestrel h2 keepalive (`Host/SiteKestrelSetup.cs` — server-wide, also governs the metrics port); (3) the server's single-stream guard is now **newest-wins with an awaited handover** — a fresh authenticated `Sync` cancels a dead/wedged incumbent and *waits* for it to finish (`TakeoverTimeout` 10s) before starting, so a half-open link can no longer wedge the pair behind `AlreadyExists` while the zombie side reports `LocalDbReplicationConnected=true`.
- **LocalDb 0.3.3 (2026-08-20, ScadaBridge arch-review P4-7; wire-compatible, no schema change — superseded by 0.3.4 below in the same bullet):** two latent `LwwApplier` hazards closed before any table could hit them. (1) **`RegisterReplicated` now REFUSES a table that is the parent of a foreign key with a row-mutating `ON DELETE` action (`CASCADE`/`SET NULL`/`SET DEFAULT`) from any other table** (`LocalDbRegistrationException` naming `child.column ON DELETE <action>`; `RESTRICT`/`NO ACTION` still allowed) — the applier's `INSERT OR REPLACE` runs with `foreign_keys=ON`, so the implicit delete fired those actions while CDC capture was suppressed, i.e. child rows changed on one node and never replicated. Registration-time only: SQLite can't add an FK to an existing table, so a later-created child is caught at the next boot's registration. None of ScadaBridge's eleven tables has an FK (verified on the live rig file: zero `REFERENCES` across 29 `CREATE TABLE`s) — **keep it that way; a cascading-FK parent can never be one of `ReplicatedTables`** (register row 105). (2) The per-entry apply (delete-or-upsert + row-version ledger write) is now **atomic under a `SAVEPOINT`** — a fault between the two statements used to leave a deleted row with a live ledger entry, which `SnapshotStreamer` warns-and-skips forever; poison-catch semantics unchanged (narrow `SqliteException`, per-entry dead-letter, batch continues). Cost: two extra in-memory statements per applied entry on a backlog drain. **Both are pinned here** (2026-08-20): `SiteLocalDbReplicatedTableFkGuardTests` asserts at build time that no `ReplicatedTables` entry has an inbound row-mutating FK (register row 105), and **pre-2026-08-20 site files must be VACUUMed once offline** to get `auto_vacuum=INCREMENTAL` — `docker/vacuum-site-localdb.sh` (pair stopped together; done on all three rig pairs; topology guide → *one-off offline VACUUM*; row 104). **LocalDb 0.3.4 (2026-08-20, register row 106; wire-compatible, sender-side only):** a converged pair whose sender oplog was ack-pruned to empty used to stamp snapshots `as_of_seq 0`, so a peer with a zero watermark (a rebuilt node — the rig's site-a-a since 2026-08-15) received a FULL snapshot on every reconnect until the sender wrote again (127 k rows per restart, observed). The as-of now floors at `last_acked_seq`; proven live — second restart after the pin exchanged zero snapshots. Lesson for rebuilding a node from its peer: the first reconnect snapshots, the second must not.
- **LocalDb 0.3.3 (2026-08-20, ScadaBridge arch-review P4-7; wire-compatible, no schema change — superseded by 0.3.4 below in the same bullet):** two latent `LwwApplier` hazards closed before any table could hit them. (1) **`RegisterReplicated` now REFUSES a table that is the parent of a foreign key with a row-mutating `ON DELETE` action (`CASCADE`/`SET NULL`/`SET DEFAULT`) from any other table** (`LocalDbRegistrationException` naming `child.column ON DELETE <action>`; `RESTRICT`/`NO ACTION` still allowed) — the applier's `INSERT OR REPLACE` runs with `foreign_keys=ON`, so the implicit delete fired those actions while CDC capture was suppressed, i.e. child rows changed on one node and never replicated. Registration-time only: SQLite can't add an FK to an existing table, so a later-created child is caught at the next boot's registration. None of ScadaBridge's twelve replicated tables has an FK (verified on the live rig file: zero `REFERENCES` across 29 `CREATE TABLE`s) — **keep it that way; a cascading-FK parent can never be one of `ReplicatedTables`** (register row 105). (2) The per-entry apply (delete-or-upsert + row-version ledger write) is now **atomic under a `SAVEPOINT`** — a fault between the two statements used to leave a deleted row with a live ledger entry, which `SnapshotStreamer` warns-and-skips forever; poison-catch semantics unchanged (narrow `SqliteException`, per-entry dead-letter, batch continues). Cost: two extra in-memory statements per applied entry on a backlog drain. **Both are pinned here** (2026-08-20): `SiteLocalDbReplicatedTableFkGuardTests` asserts at build time that no `ReplicatedTables` entry has an inbound row-mutating FK (register row 105), and **pre-2026-08-20 site files must be VACUUMed once offline** to get `auto_vacuum=INCREMENTAL` — `docker/vacuum-site-localdb.sh` (pair stopped together; done on all three rig pairs; topology guide → *one-off offline VACUUM*; row 104). **LocalDb 0.3.4 (2026-08-20, register row 106; wire-compatible, sender-side only):** a converged pair whose sender oplog was ack-pruned to empty used to stamp snapshots `as_of_seq 0`, so a peer with a zero watermark (a rebuilt node — the rig's site-a-a since 2026-08-15) received a FULL snapshot on every reconnect until the sender wrote again (127 k rows per restart, observed). The as-of now floors at `last_acked_seq`; proven live — second restart after the pin exchanged zero snapshots. Lesson for rebuilding a node from its peer: the first reconnect snapshots, the second must not.
- **LocalDb 0.2.1** (two adversarial-review findings against 0.2.0, wire-compatible, no schema change) fixes: (1) `DeregisterReplicated` was deleting the HLC clock's only durable crash anchor along with the table rows — an ungraceful exit (crash, SIGKILL, power loss) on a fully-deregistered file left nothing to recover from, so the reopened clock could restart from wall-clock time and re-issue stamps below HLCs peers already hold (silently discarded by LWW); the current clock is now flushed inside the deregistration transaction, before the prune. (2) The sync inbox was an **unbounded** channel, so inbound memory was a function of what the peer sends — a mutual full snapshot (both sides of a pair baseline unconditionally when replication is enabled late) buffered whole, in memory, on both nodes at once; it is now **bounded at 64 messages** with wait-mode backpressure (unrelated to, and not to be confused with, the rig's separate `MaxBatchSize: 64` DB-read-page pin above), which required moving the snapshot to run alongside the receive loop (not inside the pre-loop handshake) so the inbox is drained for the whole session and can't deadlock itself.
- All timestamps are UTC throughout the system.
- Inter-cluster communication uses **three** transports, not two — **all cross-cluster command/control and data now rides gRPC** after the ClusterClient→gRPC migration's Phase 4 (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`) deleted Akka `ClusterClient`/`ClusterClientReceptionist`: (1) **gRPC command/control** — site→central over the central-hosted `CentralControlService` (`GrpcCentralTransport`, sticky central-a→central-b channel pair; deployments/notifications/health/heartbeat/audit-ingest/reconcile), and central→site over the site-hosted `SiteCommandService` (`GrpcSiteTransport`, per-site NodeA→NodeB channel pair; the 28 lifecycle/OPC-UA/query/parked/route/failover commands); (2) **gRPC** server-streaming for real-time data (attribute values, alarm states, `SiteStreamService`); and (3) **plain token-gated HTTP** for the deployment config itself — notify-and-fetch, the site pulls the config from `DeploymentConfigEndpoints` (`ManagementService/DeploymentConfigEndpoints.cs`) with an `X-Deployment-Token` header, `AllowAnonymous` with the per-deployment token as the entire security boundary. The gRPC boundary is per-site PSK-authenticated (`ControlPlaneAuthInterceptor`, unchanged). There is **no receptionist registration** — discovery is by dialling configured endpoints; central builds one `SitePairChannelProvider` per site (addresses from `Site.GrpcNodeAAddress`/`GrpcNodeBAddress`, refreshed from the DB every 60s and on admin changes), sites dial `ScadaBridge:Communication:CentralGrpcEndpoints` (both central nodes, h2c on `CentralGrpcPort` 8083, **NOT** via Traefik). **Discovery is asymmetric by design:** central discovers site gRPC addresses from the *database* (refreshable at runtime), sites discover central from *appsettings* (`CentralGrpcEndpoints`, static — restart required; `StartupValidator` requires a Site node to list at least one). `Akka.Cluster.Tools` stays for ClusterSingleton; only the ClusterClient part is gone. **Central never buffers for an unreachable site** — the send fails with the caller's Ask/deadline timing out; a `ConnectionStateChanged` mechanism built for this was deleted as dead code.
@@ -141,7 +141,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
- Per-instance operation lock covers all mutating commands (deploy, disable, enable, delete).
- **Bulk site deployment (`DeploymentService.DeploySiteAsync`, arch-review WP2.5)** redeploys every out-of-date instance at a site through one flatten session (a template chain shared by N instances is walked once) in three phases — serial Prepare, bounded-parallel Send (`SiteDeploymentMaxParallelism`, default 4), serial Finalize — still **not all-or-nothing**: any one instance's failure doesn't block the rest. **Disabled instances are SKIPPED by bulk deploy as of 2026-08-19 (arch-review P7-2)** — they are reported in the summary's additive `SkippedInstances` list, never flattened/sent/re-enabled (single-instance deploy remains the explicit operator path that deploys+enables a Disabled instance); **Up-to-date instances are ALSO skipped as of 2026-08-20 (arch-review P7-6)** — a redeploy is not idempotent at the site (it resets static overrides and clears native alarm state), so `DeploySiteAsync` pre-filters through `IStaleInstanceProbe` (trustworthy since P7-4 fixed its memo timing) and reports unchanged instances in `SkippedInstances` with reason `up-to-date`; the filter is conservative (every uncertainty deploys: missing snapshot, `ShouldQuerySiteBeforeRedeploy`, null/faulting probe). Additive `MgmtDeploySiteCommand.Force` / CLI `deploy site --force` restores the unconditional push for what central cannot see (restored volume, rebuilt node) — **Disabled still wins under `--force`**. Also P7-5: the three phases share one scoped DbContext and a failed `SaveChanges` leaves the entity tracked, so one permanent failure used to cascade to every later instance and strand records `InProgress`; `MarkDeploymentFailedAsync` now detaches pending changes and retries the `Failed` write once on the clean tracker; Prepare's per-instance reconcile also runs under a short `SiteDeploymentReconcileTimeout` (5s) with a once-per-batch unreachable-site circuit-breaker, so a down site no longer costs 30s × N serial before the first send (P7-3). The `PendingDeployment` staging row (5-minute TTL) is written per-instance immediately before that instance's round-trip in the Send phase, not upfront in Prepare, so a large batch's tail instances don't have their TTL clock already half-spent by the time they're reached.
- **Site artifact revision + fleet-wide pending-updates indicator (2026-09-14, `docs/plans/2026-09-14-pending-updates-design.md`).** `Site.ArtifactRevisionHash`/`ArtifactsDeployedAt` (migration `AddSiteArtifactRevision`) are stamped by `ArtifactDeploymentService` only on a site whose artifact deploy succeeds, so a redeployed fleet's artifacts are trackable per site the same way instance staleness already is. `PendingUpdatesEvaluator` combines that with `DeploymentService.GetInstanceStalenessAsync` (one shared `FlattenSession` for the whole fleet) into a `PendingUpdatesSnapshot`, held behind a process-wide singleton `IPendingUpdatesCache` (`ScadaBridge:DeploymentManager:PendingUpdatesCacheTtl`, default 60s) so N Blazor circuits cost one fleet flatten per TTL, not one each. Surfaces: the Sites page's per-site **Artifacts** badge (`Current`/`Out of date`/`Never deployed`) + instance-staleness line + header **Re-check**, and a rail-footer **pending updates** indicator (Deployer/Administrator only, 30s poll, site-scoped). **Upgrade consequence:** every site reads *Never deployed* until one artifact deploy runs post-upgrade — deliberate, not a bug. Also shipped: the rail nav now sits in its own scroll pane (`NavMenu`'s `.rail-scroll` + `site.css`) so a long nav can't push the footer/indicator off-screen; the `ZB.MOM.WW.Theme` kit itself is untouched (register row 123 proposes upstreaming it).
- **Site secrets ride the artifact deploy; the gRPC secrets hub is retired in ScadaBridge (2026-09-28, `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`).** `Deploy Artifacts` now ships each site **only the secrets it references** — `SiteSecretsCollector` scans a site's data-connection JSON for `secret:<name>` (`SecretReferenceScanner`, protocol-agnostic, bootstrap names like `SB-GRPC-PSK-*` excluded) and resolves each against the central `ISecretStore`; a missing/tombstoned secret fails **that site's** deploy before anything is sent (fail-closed), ciphertext only ever crosses the wire, and secrets factor into `ArtifactRevisionHashService`'s staleness hash (a missing secret hashes as a distinguishable marker, so an unresolvable reference never reads *Current*; a site with no secret references hashes identically whether the field is null or empty). A pre-feature site (answers with no `SecretsApplied`) or a partial apply is also treated as a deploy failure and is never stamped; one `DeploySiteSecrets`/`SiteSecrets` audit row (`secrets` name + revision, `missing` and `removed` names, never ciphertext — `removed` comes from the site's `SecretsRemoved` reply, so a removal-only deploy is audited too) is written per site deploy that carried, lacked or retired secrets, on both the sweep and the per-site retry paths. A central secret-store read failure fails only the sites that reference a secret and leaves them *Out of date* in the pending-updates check — it no longer aborts the sweep or the fleet-wide check — and both artifact-deploy CLI commands exit 1 when any site failed. Site-side: `deployed_secrets` is the 11th LocalDb-replicated table (ciphertext rows only, upsert-then-soft-remove); the per-node `DeployedSecretsMirror` (`SiteRuntime:DeployedSecretsMirrorInterval`, default 15s, also run synchronously right after an active-node apply) reconciles it into that node's local `ISecretStore`, ledgers what it mirrored in the node-local `mirrored_secrets` table so an out-of-band local secret sharing a deployed name is never silently touched (a mirror-owned name deleted or edited locally is healed back to the deployed value on the next pass, even at an unchanged revision), and trial-decrypts every pass to catch a KEK mismatch. Secrets are mirrored before the deploy's data connections reach the DCL; a rotation with unchanged connection config takes effect on the next (re)connect. **A site pair using deployed secrets must run LocalDb replication** — otherwise only the node active at deploy time holds them. Central runs one shared secret store, `Secrets:Store:Provider=SqlServer` (Central role only; a clustered central left on `Sqlite` gets a boot Warning) — the pre-host `${secret:}` expander reads the same store the running node serves. **ALL secrets replication is retired in ScadaBridge** — the pull-only gRPC hub and the SQL-Server replication mode both — `Secrets:Replication:Enabled` set to anything but empty/`false` now fails `StartupValidator` on either role with a message pointing at Deploy Artifacts / `Secrets:Store:Provider=SqlServer`.
- **Site secrets ride the artifact deploy; the gRPC secrets hub is retired in ScadaBridge (2026-09-28, `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`).** `Deploy Artifacts` now ships each site **only the secrets it references** — `SiteSecretsCollector` scans a site's data-connection JSON for `secret:<name>` (`SecretReferenceScanner`, protocol-agnostic, bootstrap names like `SB-GRPC-PSK-*` excluded) and resolves each against the central `ISecretStore`; a missing/tombstoned secret fails **that site's** deploy before anything is sent (fail-closed), ciphertext only ever crosses the wire, and secrets factor into `ArtifactRevisionHashService`'s staleness hash (a missing secret hashes as a distinguishable marker, so an unresolvable reference never reads *Current*; a site with no secret references hashes identically whether the field is null or empty). A pre-feature site (answers with no `SecretsApplied`) or a partial apply is also treated as a deploy failure and is never stamped; one `DeploySiteSecrets`/`SiteSecrets` audit row (`secrets` name + revision, `missing` and `removed` names, never ciphertext — `removed` comes from the site's `SecretsRemoved` reply, so a removal-only deploy is audited too) is written per site deploy that carried, lacked or retired secrets, on both the sweep and the per-site retry paths. A central secret-store read failure fails only the sites that reference a secret and leaves them *Out of date* in the pending-updates check — it no longer aborts the sweep or the fleet-wide check — and both artifact-deploy CLI commands exit 1 when any site failed. Site-side: `deployed_secrets` is the 11th LocalDb-replicated table (ciphertext rows only, upsert-then-soft-remove); the per-node `DeployedSecretsMirror` (`SiteRuntime:DeployedSecretsMirrorInterval`, default 15s, also run synchronously right after an active-node apply) reconciles it into that node's local `ISecretStore`, ledgers what it mirrored in the node-local `mirrored_secrets` table so an out-of-band local secret sharing a deployed name is never silently touched (a mirror-owned name deleted or edited locally is healed back to the deployed value on the next pass, even at an unchanged revision), and trial-decrypts every pass to catch a KEK mismatch. Each node also records its current mirror failures (names + messages only) in the 12th replicated table `secret_mirror_status` — written only when its failure set changes, plus once per start-up — and the active node forwards every node's row as `SiteHealthReport.SecretMirrorStatuses` (proto field 33), so a **standby** KEK mismatch shows as a *Secret mirror failing* badge on the Health dashboard instead of only in the standby's log (review finding M1). Secrets are mirrored before the deploy's data connections reach the DCL; a rotation with unchanged connection config takes effect on the next (re)connect. **A site pair using deployed secrets must run LocalDb replication** — otherwise only the node active at deploy time holds them. Central runs one shared secret store, `Secrets:Store:Provider=SqlServer` (Central role only; a clustered central left on `Sqlite` gets a boot Warning) — the pre-host `${secret:}` expander reads the same store the running node serves. **ALL secrets replication is retired in ScadaBridge** — the pull-only gRPC hub and the SQL-Server replication mode both — `Secrets:Replication:Enabled` set to anything but empty/`false` now fails `StartupValidator` on either role with a message pointing at Deploy Artifacts / `Secrets:Store:Provider=SqlServer`.
- **Create Instance picks its template from the Templates folder tree** (2026-08-21): the flat `<select>` is gone. The Template field is a read-only folder-qualified path box (`TemplatePathFormatter`, "Folder / Sub / Name") + a **Choose…** button, both opening `TemplatePickerDialog` — an `IDialogService.ShowAsync` body wrapping the **same shared `TemplateFolderTree`** the Templates page renders, with its own search, expand/collapse-all, templates-only selection and double-click-to-pick. It opens on the current pick via the tree's additive **`RevealKeys`** parameter (ancestors of the named `f:`/`t:` keys start expanded when no `Filter` is active; an active filter's reveal set still wins; callers passing nothing are byte-identical). `InstanceCreate` gained `[SupplyParameterFromQuery] TemplateId`, which the Templates tree's new **Deployer-gated "New Instance…"** context-menu item navigates with; combined with `?siteId=` it populates the Connection Bindings card on first render. **The trap for anyone extending the picker:** in `TreeViewSelectionMode.Single` the inner `TreeView` reads the *singular* `SelectedKey`/`SelectedKeyChanged` pair, but `TemplateFolderTree` forwards only the plural (Checkbox-mode) `SelectedKeys` pair — so a Single-mode caller cannot get a row highlight through the shared tree and must own selection in its own `NodeContent`, as the picker does. Design: `docs/plans/2026-08-21-template-picker-design.md`.
- **Create Instance binds module groups at creation** (2026-08-21): the create page's Connection Bindings card flattens template+site through the additive `IFlatteningPipeline.PreviewAsync` (an unsaved instance — no validation, no hash, nothing persisted, instance repo never touched) and reuses `BindingGroupBuilder`; `NodeBrowserDialog.SelectableNodeClasses` lets it pick **Object** nodes (default stays Variable-only for every existing caller). Create = `CreateInstanceAsync` → `SetConnectionBindingsAsync` (the existing upsert, no new server surface) → always navigate to Configure with `?created=1&bound=N|&bindError=…`, because the row exists whether or not the second call succeeded. Partial binding is legal: a group missing a connection or an Object is skipped and stays unbound. Design: `docs/plans/2026-08-21-create-instance-bindings-design.md`.
- Site-side apply is all-or-nothing per instance.
@@ -175,7 +175,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
- One row per lifecycle event; cached calls produce 4+ rows per operation (`Submitted`, `Forwarded`, `Attempted`, `Delivered`/`Parked`/`Discarded`).
- `ExecutionId` (`uniqueidentifier NULL`) is the universal per-run correlation value — every audit row emitted by one script execution / inbound request shares it; `CorrelationId` remains the per-operation lifecycle id (NULL for sync one-shots).
- `ParentExecutionId` (`uniqueidentifier NULL`) is the cross-execution spawn pointer — every row of a spawned run carries the spawner's `ExecutionId`; bridges inbound API → routed-site-script, alarm-triggered on-trigger scripts, and nested `CallScript`/`CallShared` invocations; `IX_AuditLog_ParentExecution` backs the filter + the recursive execution-tree walk. **Tag-cascade (alarm leg) is populated, not just plumbed** — M5.4 T4 threaded the `parentExecutionId` parameter but every `AlarmActor.SpawnAlarmExecution` call site passed null, so alarm runs were silently always roots; the id now rides site-locally as `SetStaticAttributeCommand.SourceExecutionId` → `AttributeValueChanged.SourceExecutionId` → `SpawnAlarmExecution` (all additive/nullable, no wire/proto/schema change; `Expression` triggers capture the writer *with* the evaluated snapshot since the eval completes off-dispatcher). Sources are `ScriptRuntimeContext.SetAttribute` (the run's own `ExecutionId`) and `Route.To(...).SetAttributes(...)` (the inbound request's). **Still roots by design, not omission:** alarms fired by DCL data (external values have no spawning execution — including the device echo of a script write to a *data-sourced* attribute, so only **static** writes cascade), and `ScriptActor` value-change/conditional/expression/timer trigger runs (a timer tick has no spawner; a `WhileTrue`/interval run has no single identifiable write).
- Site SQLite hot-path first, then gRPC telemetry to central; ingest is idempotent on `EventId`; periodic reconciliation pull as fallback when telemetry is lost. **The site audit DB is a separate, unreplicated SQLite file — NOT one of LocalDb's eleven consolidated tables** — at the **required** `AuditLog:SiteWriter:DatabasePath` (arch-review WP1.2, finding #2 High), `/app/data/auditlog.db` on the rig; `StartupValidator` refuses to boot a Site node without it. `DatabasePath` used to default to CWD-relative `auditlog.db`, which on the docker rig landed on the container's ephemeral overlay, not the mounted volume — the same data-loss pattern the pre-LocalDb legacy S&F/tracking databases hit (see the LocalDb bullet above), independently rediscovered here because this file sits outside LocalDb's management. The writer loop now also honors `FlushIntervalMs` (was validated but never read — one fsync'ing commit per event even at trickle rate) and sets `PRAGMA synchronous = NORMAL` (audit is best-effort by design, so the narrower power-loss window is an acceptable trade for far fewer fsyncs; WAL still guarantees no corruption).
- Site SQLite hot-path first, then gRPC telemetry to central; ingest is idempotent on `EventId`; periodic reconciliation pull as fallback when telemetry is lost. **The site audit DB is a separate, unreplicated SQLite file — NOT one of LocalDb's twelve consolidated tables** — at the **required** `AuditLog:SiteWriter:DatabasePath` (arch-review WP1.2, finding #2 High), `/app/data/auditlog.db` on the rig; `StartupValidator` refuses to boot a Site node without it. `DatabasePath` used to default to CWD-relative `auditlog.db`, which on the docker rig landed on the container's ephemeral overlay, not the mounted volume — the same data-loss pattern the pre-LocalDb legacy S&F/tracking databases hit (see the LocalDb bullet above), independently rediscovered here because this file sits outside LocalDb's management. The writer loop now also honors `FlushIntervalMs` (was validated but never read — one fsync'ing commit per event even at trickle rate) and sets `PRAGMA synchronous = NORMAL` (audit is best-effort by design, so the narrower power-loss window is an acceptable trade for far fewer fsyncs; WAL still guarantees no corruption).
- Cached operations: site emits a single additively-extended `CachedCallTelemetry` packet carrying both audit events and operational state; central writes `AuditLog` + `SiteCalls` in one transaction.
- Payload cap 8 KB by default / 64 KB on error rows; auth headers redacted by default; SQL parameter values captured by default; per-target redaction opt-in. Inbound API: full verbatim capture up to `InboundMaxBytes` (default 1 MiB); request headers stored in `Extra.requestHeaders` (post-redaction); per-method `SkipBodyCapture` flag suppresses bodies while still recording headers + metadata; `AuditInboundCeilingHits` counter surfaced on health snapshot. (M5.3 T7)
- Audit-write failure NEVER aborts the user-facing action — audit is best-effort, the action's own success/failure path is authoritative.
docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md
+9
View File
@@ -262,6 +262,15 @@ accurate as the spec:
never deletes a name it did not ledger, so they were removed once by hand on the rig (both
nodes stopped together; see the live-gate doc). QA and vd03 never ran the hub
(`Secrets:Replication:Enabled=false`), so nothing like this exists there.
- **Standby mirror failures reach central (follow-up, review finding M1).** A `DeployedSecretsMirror`
failure on the standby (e.g. a KEK mismatch) was only logged on that node — site health reports
are sent by the active node only. Each node now records its current failure set (names +
messages, never values) in a new, **12th LocalDb-replicated table `secret_mirror_status`** (one
row per node keyed by `node_name`), written only when the set changes plus once per start-up; a
per-node `SecretMirrorStatusReporter` puts every node's row on the active node's report as
`SiteHealthReport.SecretMirrorStatuses` (proto `SiteHealthReportDto` field 33, presence-wrapped),
and the Central UI health card shows a *Secret mirror failing* badge + per-node list. Requires
LocalDb replication for the standby's row to be visible, the same prerequisite as the secrets.
- **Rig posture.** `docker/docker-compose.yml`'s central nodes now carry
`Secrets__Store__Provider=SqlServer` plus a shared KEK; site-a shares that KEK (it is the
secrets-via-artifact-deploy site); site-b/site-c deliberately carry a **mismatched** KEK so the
docs/requirements/Component-CentralUI.md
+1
View File
@@ -291,6 +291,7 @@ Per-leaf alarm rendering (leaf nodes are individual conditions for native alarms
- Headline **Notification Outbox KPI tiles** — queue depth, stuck count, and parked count. These are central-computed by the Notification Outbox from the central `Notifications` table (not part of any site health report). The full outbox view is on the dedicated Notification Outbox page.
- Headline **Site Call Audit KPI tiles** — buffered count, parked count, and failed-last-interval. These are central-computed by the Site Call Audit component from the central `SiteCalls` table (not part of any site health report). The full cached-call view is on the dedicated Site Calls page.
- Headline **Audit KPI tiles** — three tiles in a new "Audit" KPI group: **Audit volume**, **Audit error rate**, and **Audit backlog**. These are sourced from the Audit Log component (#23) and Health Monitoring per the metric definitions in Component-HealthMonitoring.md; the dashboard simply surfaces them. The full audit query view is on the dedicated Audit Log page.
- **Secret mirror failing** (review finding M1) — a warning badge on an online site's card when any node's `SiteHealthReport.SecretMirrorStatuses` entry lists failures, plus a compact warning panel at the top of the card body listing *node · secret: message* for each failure. The standby's failures appear here too (they ride the active node's report), so a master-key mismatch on the standby is visible before a failover exposes it. Null (no data) and all-healthy nodes render nothing; offline sites are skipped like the other warning badges.
- **Trigger failover** (`CentralFailoverControl`, Administrator only) — on the central card and on each site card. Disabled with an explanatory tooltip when there is no standby to take over. It always confirms first, and the confirmation's red button reads **Fail over** (see the dialog-label contract under Technology); the central variant's copy states that this page is served by the node being restarted and will disconnect and reconnect, while the site variant deliberately does not, since a site is a different cluster.
- **Shared KPI cache (arch-review WP2.4).** All four KPI families above — Notification Outbox, Site Call Audit, Audit, and their per-site/per-node breakdowns — are read through a process-level `IKpiSnapshotCache`, not queried per page load. Each accessor is independently memoized (8s TTL) with single-flight production, so N Blazor circuits polling the same KPI inside one window (e.g. ten operators with the Health dashboard open) cost the node ONE aggregate SQL round trip per KPI per window, not N. A failed query is never memoized — the next caller re-attempts it, and the calling page's existing per-tile "unavailable" degradation is unchanged. Every KPI-tile page (Health, Notification Outbox, Site Calls, Audit Log) shares the same cache instance.
docs/requirements/Component-HealthMonitoring.md
+2
View File
@@ -39,6 +39,7 @@ Site clusters (metric collection and reporting). Central cluster (aggregation an
| `AuditRedactionFailure` | Audit Log (central) | Count of payload redactor errors (over-redacted payloads, safety-net hit) since the last interval. |
| `LocalDbReplicationConnected` | Consolidated site LocalDb | Whether a peer sync session is currently running on this node. **Nullable — null means "no reading", not "disconnected".** Reported only by a node with replication configured; an unconfigured node reports **null**, never `false` (register row 80 — see *Unconfigured replication reports null* below). |
| `StartupReconcilerActive` | Host (site) | Whether this node spawned its `SiteReconciliationActor` at boot. **Tri-state:** null = not reported (the central self-report, or a pre-row-73 site build), `true` = running, `false` = the node booted with no `IDeploymentConfigFetcher` and therefore has **no startup self-heal**. Latched once at boot by the Host — not polled. Drives the *No self-heal* dashboard badge (register row 73). |
| `SecretMirrorStatuses` | Site Runtime (`DeployedSecretsMirror` via the replicated `secret_mirror_status` table) | Every site node's deployed-secrets mirror status — node name, when its failure set last changed, and its current failures (`{SecretName, Message}`; names and messages only, never values). Review finding M1: the **standby's** failures ride the **active** node's report, because each node writes its own row to a LocalDb-replicated table and the active node's `SecretMirrorStatusReporter` reads them all. **Nullable — null = no data** (central self-report, a pre-M1 site, reporter not yet run); an empty list = wired, no row yet; a node with an empty `Failures` list is healthy. Point-in-time (not drained on collect). Wire: `SiteHealthReportDto.secret_mirror_statuses` (field **33**, a presence wrapper so null ≠ empty). Drives the *Secret mirror failing* dashboard badge + per-node failure list. On an unreplicated pair only the active node's own row appears. |
| `LocalDbOplogBacklog` | Consolidated site LocalDb | Unacked oplog depth (Prometheus `localdb_oplog_depth`). **Nullable — null means "no reading", NOT "connected with an empty backlog"** (a failed poll rendered as 0 would report a pair that cannot read its own oplog as perfectly healthy). **Expected transient spikes:** the daily `site_events` retention purge and the storage-cap trim are CDC-captured like any other write, so the backlog jumps by the batch size at purge time and drains as the peer acks it — see `Component-SiteEventLogging.md` → Storage and `docs/deployment/topology-guide.md` → *Reading the replication backlog* for the healthy-versus-faulty signature. |
## Reporting Protocol
@@ -232,6 +233,7 @@ endpoint (`MetricsPort`, default 8084) rather than sent to central.
- **Communication Layer**: Transports health reports from sites to central.
- **Data Connection Layer (site)**: Provides connection health metrics.
- **Site Runtime (site)**: Provides script error rate and alarm evaluation error rate metrics.
- **Site Runtime (site) — deployed secrets**: Provides `SecretMirrorStatuses` via its own `SecretMirrorStatusReporter` hosted service (SiteRuntime already references HealthMonitoring and owns the `secret_mirror_status` storage, following the `SiteStreamAlarmDropReporter` precedent).
- **Store-and-Forward Engine (site)**: Provides buffer depth metrics, including the notification backlog awaiting forward to central.
- **Cluster Infrastructure (site)**: Provides node role status.
- **Host (site)**: Supplies the two boot-time facts Health Monitoring must not derive for itself — the LocalDb replication-configured gate behind `LocalDbReplicationConnected`/`LocalDbOplogBacklog` (from `SiteLocalDbSetup.ReplicationIsConfigured`), and `StartupReconcilerActive`, latched where the Host decides whether to spawn `SiteReconciliationActor`. Both follow the `SiteEventLogReplicationCheck` precedent: the Host owns the configuration reading, the component owns the reporting.
docs/requirements/Component-SiteRuntime.md
+3 -1
View File
@@ -434,9 +434,10 @@ The Instance Actor owns native-alarm setup alongside its computed Script and Ala
## Deployed Secrets (Site SQLite / LocalDb)
Site secrets ride the artifact deploy (design `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`; see that doc's *Implementation notes (as built)* for deviations from the original design). Two tables, two different replication postures:
Site secrets ride the artifact deploy (design `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`; see that doc's *Implementation notes (as built)* for deviations from the original design). Three tables, two different replication postures:
- **`deployed_secrets`** — the **11th LocalDb-replicated table** (`SiteLocalDbSetup.ReplicatedTables`, up from ten). Columns: `name` (PK), `row_json` (the shipped `SecretArtifact` — ciphertext, nonce, tag, wrapped DEK, `KekId`, revision; never plaintext), `revision`, `updated_utc`, `is_removed` (soft-remove flag, default 0), `deployment_id`, `applied_at`. No FKs (the standing FK guard — `SiteLocalDbReplicatedTableFkGuardTests` — applies to it like every other replicated table). `UpsertDeployedSecretsAsync` is full-set upsert-then-soft-remove, never a hard `DELETE`, so a removal is itself a replicated row change the standby node can react to.
- **`secret_mirror_status`** — the **12th LocalDb-replicated table** (review finding M1). One row per site node: `node_name` (PK — `INodeIdentityProvider.NodeName`, i.e. `node-a`/`node-b`, falling back to the machine name when unconfigured), `failures_json` (a JSON array of `{SecretName, Message}` — the node's *current* mirror failures; names and messages only, never values or ciphertext), `checked_at`. No FKs. It is per-node *data* but replicated on purpose: each node writes only its own row, and replication is what lets the **active** node read the **standby's** row onto the site health report (only the active node sends one — see *Mirror status on the health report* below).
- **`mirrored_secrets`** — a **node-local, unreplicated** ledger (`name` PK, `revision`, `mirrored_at`) recording which names *this specific node* has applied into its own `ISecretStore`. Deliberately not registered with LocalDb: it is a per-node fact (which names this node mirrored), not site-wide state, and replicating it would make one node's mirroring progress masquerade as the other's.
**`DeployedSecretsMirror`** (`SiteRuntime/Secrets/DeployedSecretsMirror.cs`) is a per-node `BackgroundService` — registered on **every** site node, not a singleton — that reconciles `deployed_secrets` into the node's own local `ISecretStore` (site nodes never run a replication-decorated store; see Component-Host.md → *Secrets*):
@@ -450,6 +451,7 @@ Site secrets ride the artifact deploy (design `docs/plans/2026-09-28-site-secret
- **Isolation.** One row failing to apply/verify does not block any other row in the same pass.
- **Cache invalidation.** A successful apply or delete invalidates the shared `ISecretResolver` cache before verifying, so the verify step (and any script/connection reading the secret) sees the fresh value immediately rather than waiting out the resolver's cache TTL.
- **Never logs a secret value** — log lines and the reported `MirrorResult` carry only names, revisions, and `KekId`s.
- **Mirror status on the health report (review finding M1).** A failure on the **standby** — above all a secret that will not decrypt because the standby's master key differs from central's — used to live only in that node's log, invisible until a failover broke its connections. After every pass the mirror records its node's current failure set in `secret_mirror_status`, but **only when the set (sorted name+message pairs) differs from what this process last wrote**, plus once on the first pass after start-up so a restart refreshes a stale row — never on every 15 s tick, because every write is a replicated CDC change. The write is best-effort: a failure logs a Warning (repeats at Debug), never affects the pass result, and is retried next pass. `SecretMirrorStatusReporter` (`SiteRuntime/Secrets/`, every site node, 30 s, immediate first probe) reads every node's row into `ISiteHealthCollector.SetSecretMirrorStatuses`, so the active node's report carries both nodes' status as `SiteHealthReport.SecretMirrorStatuses` (see Component-HealthMonitoring.md). On an **unreplicated** pair only the active node's own row is visible — the same replication prerequisite deployed secrets already have.
---
src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/Health.razor
+37
View File
@@ -275,6 +275,14 @@
No self-heal
</span>
}
@if (HasSecretMirrorFailures(state))
{
<span class="badge bg-warning text-dark"
data-test="secret-mirror-badge"
title="A site node could not apply and/or decrypt one or more deployed secrets. A standby node's failures surface here before a failover would expose them — see the details in the card body.">
Secret mirror failing
</span>
}
@if (!state.IsOnline && state.LastStatusChangeAt is { } changedAt)
{
<small class="text-muted">offline since @changedAt.ToString("u")</small>
@@ -302,6 +310,26 @@
@if (state.LatestReport != null)
{
var report = state.LatestReport;
@if (HasSecretMirrorFailures(state))
{
@* Review finding M1: every site node's deployed-secrets mirror failures,
the standby's included (carried on the active node's report). *@
<div class="alert alert-warning py-2 px-3 mb-3 small" data-test="secret-mirror-failures" role="alert">
<strong>Deployed secrets failing on a site node</strong>
<ul class="mb-0 ps-3">
@foreach (var node in report.SecretMirrorStatuses!.Where(n => n.Failures.Count > 0))
{
@foreach (var failure in node.Failures)
{
<li>
<span class="font-monospace">@node.NodeName</span> &middot;
<span class="font-monospace">@failure.SecretName</span>: @failure.Message
</li>
}
}
</ul>
</div>
}
<div class="row g-3">
@* Each logical group below sits in its own card, with the group's
head promoted to a real card-header, instead of h6 rules stacked
@@ -590,6 +618,15 @@
state.IsOnline
&& state.LatestReport?.StartupReconcilerActive == false;
// Review finding M1: warn when any site node (the standby included) reports deployed
// secrets it cannot apply/decrypt. Null ("no data": central self-report, pre-M1 site) and
// nodes with an empty failure list are healthy. Offline sites are skipped like the other
// warning badges — the Offline badge is the signal that matters there.
private static bool HasSecretMirrorFailures(SiteHealthState state) =>
state.IsOnline
&& state.LatestReport?.SecretMirrorStatuses is { } statuses
&& statuses.Any(n => n.Failures.Count > 0);
// Online central nodes, from the same ClusterNodes list the Nodes column renders.
// Fewer than two means no standby, which disables the manual-failover control. This is
// a display-side guard only — AkkaManualFailoverService re-checks against live cluster
src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SecretMirrorNodeStatus.cs
+26
View File
@@ -0,0 +1,26 @@
namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
/// <summary>
/// One site node's latest deployed-secrets mirror outcome, as carried on
/// <see cref="SiteHealthReport.SecretMirrorStatuses"/> (review finding M1).
/// </summary>
/// <remarks>
/// Every site node runs its own <c>DeployedSecretsMirror</c> and records its current
/// failure set in the LocalDb-replicated <c>secret_mirror_status</c> table, keyed by node
/// name; the ACTIVE node reads every row and ships them on its health report, so a
/// STANDBY node's failures (most importantly a master-key mismatch that stops a deployed
/// secret decrypting there) reach central before a failover exposes them. Carries names
/// and human-readable messages only — never a secret value or ciphertext.
/// </remarks>
/// <param name="NodeName">The reporting node's semantic name (<c>node-a</c>/<c>node-b</c>), or its host name when unconfigured.</param>
/// <param name="CheckedAtUtc">When that node last recorded a CHANGE in its failure set (or its first pass after start-up) — not the time of its latest pass.</param>
/// <param name="Failures">That node's current mirror failures; empty when the node's mirror is healthy.</param>
public sealed record SecretMirrorNodeStatus(
string NodeName,
DateTimeOffset CheckedAtUtc,
IReadOnlyList<SecretMirrorFailure> Failures);
/// <summary>A single deployed secret that failed to apply and/or decrypt on one site node.</summary>
/// <param name="SecretName">The secret's name.</param>
/// <param name="Message">A human-readable failure message — never includes the secret's value.</param>
public sealed record SecretMirrorFailure(string SecretName, string Message);
src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SiteHealthReport.cs
+14
View File
@@ -173,6 +173,20 @@ public record SiteHealthReport(
/// </para>
/// </remarks>
public bool? StartupReconcilerActive { get; init; }
/// <summary>
/// Every site node's latest deployed-secrets mirror outcome (review finding M1) — one
/// entry per node that has recorded a row in the replicated <c>secret_mirror_status</c>
/// table, including the STANDBY node, whose own health report is never sent.
/// </summary>
/// <remarks>
/// <b>Null = no data</b>: a central self-report, a pre-M1 site build, a site whose
/// reporter has not run yet, or a failed read. An EMPTY list means the feature is wired
/// but no node has recorded a status yet. A node entry with an empty
/// <see cref="SecretMirrorNodeStatus.Failures"/> list is a healthy mirror. Names and
/// messages only — never secret values or ciphertext.
/// </remarks>
public IReadOnlyList<SecretMirrorNodeStatus>? SecretMirrorStatuses { get; init; }
}
/// <summary>
src/ZB.MOM.WW.ScadaBridge.Communication/CentralControlGrpc/CentralControl.cs
+841 -78
View File
File diff suppressed because it is too large Load Diff
src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/CentralControlDtoMapper.cs
+40
View File
@@ -559,6 +559,32 @@ public static class CentralControlDtoMapper
dto.SiteAuditBacklog = snapshot;
}
// Review finding M1: wrapper presence keeps null ("no data") distinct from empty.
if (msg.SecretMirrorStatuses is { } mirrorStatuses)
{
var wrapper = new SecretMirrorStatusListDto();
foreach (var node in mirrorStatuses)
{
var nodeDto = new SecretMirrorNodeStatusDto
{
NodeName = node.NodeName,
CheckedAtUtc = Timestamp.FromDateTimeOffset(node.CheckedAtUtc),
};
foreach (var failure in node.Failures)
{
nodeDto.Failures.Add(new SecretMirrorFailureDto
{
SecretName = failure.SecretName,
Message = failure.Message,
});
}
wrapper.Nodes.Add(nodeDto);
}
dto.SecretMirrorStatuses = wrapper;
}
return dto;
}
@@ -620,6 +646,19 @@ public static class CentralControlDtoMapper
OnDiskBytes: snapshot.OnDiskBytes);
}
List<SecretMirrorNodeStatus>? mirrorStatuses = null;
if (dto.SecretMirrorStatuses is { } mirrorWrapper)
{
mirrorStatuses = new List<SecretMirrorNodeStatus>(mirrorWrapper.Nodes.Count);
foreach (var node in mirrorWrapper.Nodes)
{
mirrorStatuses.Add(new SecretMirrorNodeStatus(
node.NodeName,
node.CheckedAtUtc?.ToDateTimeOffset() ?? DateTimeOffset.MinValue,
node.Failures.Select(f => new SecretMirrorFailure(f.SecretName, f.Message)).ToList()));
}
}
return new SiteHealthReport(
SiteId: dto.SiteId,
SequenceNumber: dto.SequenceNumber,
@@ -657,6 +696,7 @@ public static class CentralControlDtoMapper
DetachedScriptThreads = dto.DetachedScriptThreads,
ScriptRunShedCount = dto.ScriptRunShedCount,
SiteStreamAlarmDropCount = dto.SiteStreamAlarmDropCount,
SecretMirrorStatuses = mirrorStatuses,
};
}
src/ZB.MOM.WW.ScadaBridge.Communication/Protos/central_control.proto
+23
View File
@@ -193,6 +193,25 @@ message NodeStatusListDto {
repeated NodeStatusDto nodes = 1;
}
// Review finding M1: one site node's deployed-secrets mirror failures. Names and
// human-readable messages only — never a secret value or ciphertext.
message SecretMirrorFailureDto {
string secret_name = 1;
string message = 2;
}
message SecretMirrorNodeStatusDto {
string node_name = 1;
google.protobuf.Timestamp checked_at_utc = 2;
repeated SecretMirrorFailureDto failures = 3;
}
// Presence wrapper (same reason as NodeStatusListDto): absent = null ("no data"),
// present-but-empty = the feature is wired and no node has recorded a status yet.
message SecretMirrorStatusListDto {
repeated SecretMirrorNodeStatusDto nodes = 1;
}
// Site -> Central: periodic site health report. Mirrors Commons SiteHealthReport.
// Additive-only evolution: field numbers are never reused.
message SiteHealthReportDto {
@@ -245,6 +264,10 @@ message SiteHealthReportDto {
// dedicated alarm publish queue because it was at capacity. Point-in-time
// (not reset on collect), like site_event_log_write_failures.
int64 site_stream_alarm_drop_count = 32;
// Review finding M1: every site node's deployed-secrets mirror status (the standby's
// included — it is read from a LocalDb-replicated table on the active node).
// Absent means null ("no data": central self-report, pre-M1 site, or not read yet).
SecretMirrorStatusListDto secret_mirror_statuses = 33;
}
// Central -> Site: health report ack, so a lost report is observable.
src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ISiteHealthCollector.cs
+13
View File
@@ -198,6 +198,19 @@ public interface ISiteHealthCollector
// Default no-op so test fakes do not need to be updated.
}
/// <summary>
/// Replace the latest per-node deployed-secrets mirror statuses (review finding M1) used
/// by the <c>SecretMirrorStatuses</c> field of the next <see cref="CollectReport"/> call.
/// Refreshed periodically by the site-side <c>SecretMirrorStatusReporter</c> from the
/// LocalDb-replicated <c>secret_mirror_status</c> table, so the ACTIVE node's report
/// carries the STANDBY node's mirror failures too. Point-in-time: NOT reset on collect.
/// </summary>
/// <param name="statuses">Every node's row, or <see langword="null"/> for "no data".</param>
void SetSecretMirrorStatuses(IReadOnlyList<SecretMirrorNodeStatus>? statuses)
{
// Default no-op so test fakes do not need to be updated.
}
/// <summary>
/// Replace the latest script-execution-scheduler gauges (queue depth, busy
/// thread count, and age in seconds of the oldest in-flight script) used by
src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthCollector.cs
+13 -1
View File
@@ -32,6 +32,10 @@ public class SiteHealthCollector : ISiteHealthCollector
// -1 = reconciler SKIPPED (no IDeploymentConfigFetcher)
// Null must stay distinct from -1: "we don't know" is not "self-heal is missing".
private volatile int _startupReconcilerActive;
// Review finding M1: every site node's deployed-secrets mirror status, refreshed by the
// SiteRuntime SecretMirrorStatusReporter. Null = the reporter has not run (or its read
// failed before it ever succeeded) — "no data", distinct from an empty list.
private volatile IReadOnlyList<SecretMirrorNodeStatus>? _secretMirrorStatuses;
private readonly ConcurrentDictionary<string, ConnectionHealth> _connectionStatuses = new();
private readonly ConcurrentDictionary<string, TagResolutionStatus> _tagResolutionCounts = new();
private readonly ConcurrentDictionary<string, string> _connectionEndpoints = new();
@@ -130,6 +134,12 @@ public class SiteHealthCollector : ISiteHealthCollector
_startupReconcilerActive = active ? 1 : -1;
}
/// <inheritdoc />
public void SetSecretMirrorStatuses(IReadOnlyList<SecretMirrorNodeStatus>? statuses)
{
_secretMirrorStatuses = statuses?.ToList();
}
/// <inheritdoc />
public void UpdateConnectionHealth(string connectionName, ConnectionHealth health)
{
@@ -333,7 +343,9 @@ public class SiteHealthCollector : ISiteHealthCollector
SiteStreamAlarmDropCount = Interlocked.Read(ref _siteStreamAlarmDropCount),
// Tri-state: 0 stays null ("not reported"), which must never collapse into
// false ("self-heal is missing"). Register row 73.
StartupReconcilerActive = startupReconciler == 0 ? null : startupReconciler > 0
StartupReconcilerActive = startupReconciler == 0 ? null : startupReconciler > 0,
// M1: point-in-time, not drained; null stays null ("no data").
SecretMirrorStatuses = _secretMirrorStatuses
};
}
}
src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs
+12 -3
View File
@@ -48,7 +48,7 @@ namespace ZB.MOM.WW.ScadaBridge.Host;
public static class SiteLocalDbSetup
{
/// <summary>
/// The eleven tables that participate in replication — registered when this node has a peer,
/// The twelve tables that participate in replication — registered when this node has a peer,
/// and deregistered (stale triggers dropped) when it does not.
/// </summary>
/// <remarks>
@@ -69,6 +69,13 @@ public static class SiteLocalDbSetup
/// <c>deployed_secrets</c> row, and replicating the ledger would make one node's progress
/// masquerade as the other's.
/// </para>
/// <para>
/// <c>secret_mirror_status</c> is the twelfth entry (review finding M1). It is per-node
/// DATA — one row per node, keyed by <c>node_name</c>, each written only by its own node —
/// but it is replicated precisely so the active node can read the standby's row and put
/// it on the site health report (only the active node sends one). Explicit primary key,
/// no FK, no BLOB columns.
/// </para>
/// </remarks>
internal static readonly string[] ReplicatedTables =
[
@@ -83,6 +90,7 @@ public static class SiteLocalDbSetup
"data_connection_definitions",
"native_alarm_state",
"deployed_secrets",
"secret_mirror_status",
];
/// <summary>
@@ -140,8 +148,9 @@ public static class SiteLocalDbSetup
// ELEVENTH entry to ReplicatedTables on an already-replicating site streams a
// snapshot of only that one table, where through 0.2.x the flag was a single
// per-database bit and the same edit would have re-streamed all eleven in full, in
// both directions. Nothing about the first boot changes: enabling replication on a
// site that has been running without it seeds all eleven at once, the flagged set is
// both directions (the same holds for secret_mirror_status, the twelfth, added by
// review finding M1). Nothing about the first boot changes: enabling replication on a
// site that has been running without it seeds all twelve at once, the flagged set is
// therefore every registered table, and the library sends that as an ordinary full
// snapshot. A hypothetical "register the whole list in one call" API would silently
// give the coarse behaviour back.
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageSchema.cs
+17 -3
View File
@@ -3,8 +3,8 @@ using Microsoft.Data.Sqlite;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
/// <summary>
/// DDL for the site's eleven configuration tables (nine original plus
/// <c>deployed_secrets</c> and <c>mirrored_secrets</c>), extracted from
/// DDL for the site's twelve configuration tables (nine original plus
/// <c>deployed_secrets</c>, <c>mirrored_secrets</c> and <c>secret_mirror_status</c>), extracted from
/// <see cref="SiteStorageService"/> so it can be applied by whoever owns the database
/// file.
/// <para>
@@ -23,7 +23,7 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
public static class SiteStorageSchema
{
/// <summary>
/// Creates the eleven site configuration tables when absent, and additively upgrades
/// Creates the twelve site configuration tables when absent, and additively upgrades
/// tables created by an older build. Idempotent — safe to run on every startup.
/// </summary>
/// <param name="connection">An open connection to the database that should hold the tables.</param>
@@ -136,6 +136,20 @@ public static class SiteStorageSchema
revision INTEGER NOT NULL,
mirrored_at TEXT NOT NULL
);
-- Per-node deployed-secrets mirror status (review finding M1): one row per site
-- node, keyed by node name, holding that node's CURRENT mirror failures as a JSON
-- array of {SecretName, Message} (names and messages only — never values or
-- ciphertext). Unlike mirrored_secrets this IS replicated
-- (SiteLocalDbSetup.ReplicatedTables): each node writes only its own row, and the
-- replication is exactly what lets the ACTIVE node's health report carry the
-- STANDBY node's failures to central. Written only when a node's failure set
-- changes (plus once per start-up), never per mirror tick. No foreign keys.
CREATE TABLE IF NOT EXISTS secret_mirror_status (
node_name TEXT PRIMARY KEY,
failures_json TEXT NOT NULL,
checked_at TEXT NOT NULL
);
";
command.ExecuteNonQuery();
}
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs
+82
View File
@@ -3,6 +3,7 @@ using Microsoft.Data.Sqlite;
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.LocalDb;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
@@ -924,6 +925,87 @@ public class SiteStorageService
await command.ExecuteNonQueryAsync();
}
// ── Secret mirror status (review finding M1 — LocalDb-replicated, one row per node) ──
/// <summary>
/// Records <paramref name="nodeName"/>'s current deployed-secrets mirror failures in the
/// replicated <c>secret_mirror_status</c> table (UPSERT on the node name). An empty
/// <paramref name="failures"/> list records a healthy mirror. Every call is a replicated
/// write, so callers write only when the failure set changes.
/// </summary>
/// <param name="nodeName">This node's name — the row key.</param>
/// <param name="failures">The node's current failures (names and messages only, never values).</param>
/// <param name="checkedAtUtc">When the recorded failure set was observed.</param>
/// <returns>A task that completes when the row has been stored.</returns>
public async Task SetSecretMirrorStatusAsync(
string nodeName, IReadOnlyList<SecretMirrorFailure> failures, DateTimeOffset checkedAtUtc)
{
ArgumentException.ThrowIfNullOrWhiteSpace(nodeName);
ArgumentNullException.ThrowIfNull(failures);
await using var connection = OpenConnection();
await using var command = connection.CreateCommand();
command.CommandText = @"
INSERT INTO secret_mirror_status (node_name, failures_json, checked_at)
VALUES (@nodeName, @failures, @checkedAt)
ON CONFLICT(node_name) DO UPDATE SET
failures_json = excluded.failures_json,
checked_at = excluded.checked_at";
command.Parameters.AddWithValue("@nodeName", nodeName);
command.Parameters.AddWithValue("@failures", JsonSerializer.Serialize(failures));
command.Parameters.AddWithValue("@checkedAt", checkedAtUtc.ToUniversalTime().ToString("O"));
await command.ExecuteNonQueryAsync();
}
/// <summary>
/// Returns every node's row from <c>secret_mirror_status</c>, ordered by node name. A row
/// whose <c>failures_json</c> cannot be parsed is returned with a single synthetic failure
/// saying so, rather than failing the whole read or silently reading as healthy.
/// </summary>
/// <returns>A task that resolves to every node's recorded mirror status.</returns>
public async Task<IReadOnlyList<SecretMirrorNodeStatus>> GetSecretMirrorStatusesAsync()
{
await using var connection = OpenConnection();
await using var command = connection.CreateCommand();
command.CommandText =
"SELECT node_name, failures_json, checked_at FROM secret_mirror_status ORDER BY node_name";
var results = new List<SecretMirrorNodeStatus>();
await using var reader = await command.ExecuteReaderAsync();
while (await reader.ReadAsync())
{
var nodeName = reader.GetString(0);
IReadOnlyList<SecretMirrorFailure> failures;
try
{
failures = JsonSerializer.Deserialize<List<SecretMirrorFailure>>(reader.GetString(1))
?? [];
}
catch (Exception ex) when (ex is JsonException or NotSupportedException)
{
failures =
[
new SecretMirrorFailure(
"(unreadable)",
$"Mirror status row for node '{nodeName}' could not be parsed ({ex.GetType().Name})"),
];
}
var checkedAt = DateTimeOffset.TryParse(
reader.GetString(2), null, System.Globalization.DateTimeStyles.RoundtripKind, out var parsed)
? parsed
: DateTimeOffset.MinValue;
results.Add(new SecretMirrorNodeStatus(nodeName, checkedAt, failures));
}
return results;
}
// ── Shared Script CRUD ──
/// <summary>
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Secrets/DeployedSecretsMirror.cs
+71 -1
View File
@@ -1,7 +1,9 @@
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
using ZB.MOM.WW.Secrets.Abstractions;
@@ -153,6 +155,18 @@ public sealed record MirrorResult(
/// so a LATER failure — even one with an identical fingerprint — logs at Error again. This state
/// is in-memory only (per node process), which matches the ledger's own node-local scope.
/// </para>
/// <para>
/// <b>Status row for central (review finding M1).</b> A failure on the STANDBY node used to be
/// visible only in that node's own log — site health reports are sent by the active node only.
/// After each pass this mirror records its node's current failure set (secret name + message,
/// never a value) in the LocalDb-replicated <c>secret_mirror_status</c> table, keyed by node
/// name; <c>SecretMirrorStatusReporter</c> on the active node reads every node's row onto the
/// health report. Because every write replicates via CDC, the row is written only when the
/// failure set (compared as sorted name+message pairs) differs from what this process last
/// wrote — plus once on the first pass after start-up, so a restart refreshes a stale row. The
/// write is best-effort: a failure logs a Warning (repeats at Debug) and is retried on the next
/// pass, and never affects the pass's own <see cref="MirrorResult"/>.
/// </para>
/// </remarks>
public sealed class DeployedSecretsMirror : BackgroundService
{
@@ -165,6 +179,17 @@ public sealed class DeployedSecretsMirror : BackgroundService
private readonly IOptions<SiteRuntimeOptions> _options;
private readonly ILogger<DeployedSecretsMirror> _logger;
private readonly TimeProvider _time;
private readonly string _nodeName;
/// <summary>
/// The failure set (sorted name+message pairs) this process last wrote successfully to
/// <c>secret_mirror_status</c>, or <see langword="null"/> before the first successful write
/// — which is what forces the once-per-start-up write (review finding M1).
/// </summary>
private IReadOnlyList<SecretMirrorFailure>? _lastWrittenStatus;
/// <summary>Whether a failed status write has already been logged at Warning (de-spam).</summary>
private bool _statusWriteFailureLogged;
private readonly SemaphoreSlim _gate = new(1, 1);
private bool _disposed;
@@ -199,6 +224,11 @@ public sealed class DeployedSecretsMirror : BackgroundService
/// <param name="options">Supplies <see cref="SiteRuntimeOptions.DeployedSecretsMirrorInterval"/>.</param>
/// <param name="logger">Logger instance. Never logs secret values — names, revisions, and KEK ids only.</param>
/// <param name="time">Clock driving the reconcile loop's <see cref="PeriodicTimer"/> and the tombstone-recovery stamp.</param>
/// <param name="nodeIdentity">
/// Supplies this node's name (<c>node-a</c>/<c>node-b</c>), the key of its
/// <c>secret_mirror_status</c> row. Optional: when absent or unconfigured, the machine name
/// is used instead.
/// </param>
public DeployedSecretsMirror(
SiteStorageService storage,
ISecretStore store,
@@ -206,7 +236,8 @@ public sealed class DeployedSecretsMirror : BackgroundService
ISecretCacheInvalidator? invalidator,
IOptions<SiteRuntimeOptions> options,
ILogger<DeployedSecretsMirror> logger,
TimeProvider time)
TimeProvider time,
INodeIdentityProvider? nodeIdentity = null)
{
_storage = storage ?? throw new ArgumentNullException(nameof(storage));
_store = store ?? throw new ArgumentNullException(nameof(store));
@@ -215,6 +246,7 @@ public sealed class DeployedSecretsMirror : BackgroundService
_options = options ?? throw new ArgumentNullException(nameof(options));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
_time = time ?? throw new ArgumentNullException(nameof(time));
_nodeName = nodeIdentity?.NodeName ?? Environment.MachineName;
}
/// <summary>
@@ -314,6 +346,8 @@ public sealed class DeployedSecretsMirror : BackgroundService
rows.Count(r => !r.IsRemoved) + unreadable.Count(u => !u.IsRemoved), failed.Count);
}
await RecordStatusIfChangedAsync(failedNames, failed).ConfigureAwait(false);
return result;
}
finally
@@ -322,6 +356,42 @@ public sealed class DeployedSecretsMirror : BackgroundService
}
}
/// <summary>
/// Writes this node's <c>secret_mirror_status</c> row when the pass's failure set differs
/// from the last one this process wrote (or nothing has been written yet since start-up).
/// Best-effort: never throws, so the pass result is unaffected by a status-write fault.
/// See the class remarks (review finding M1).
/// </summary>
private async Task RecordStatusIfChangedAsync(IReadOnlyList<string> failedNames, IReadOnlyList<string> failed)
{
var current = failedNames
.Select((name, i) => new SecretMirrorFailure(name, failed[i]))
.OrderBy(f => f.SecretName, StringComparer.Ordinal)
.ThenBy(f => f.Message, StringComparer.Ordinal)
.ToList();
if (_lastWrittenStatus is not null && _lastWrittenStatus.SequenceEqual(current))
{
return;
}
try
{
await _storage.SetSecretMirrorStatusAsync(_nodeName, current, _time.GetUtcNow()).ConfigureAwait(false);
_lastWrittenStatus = current;
_statusWriteFailureLogged = false;
}
catch (Exception ex)
{
_logger.Log(
_statusWriteFailureLogged ? LogLevel.Debug : LogLevel.Warning,
ex,
"Failed to record the deployed secrets mirror status for node {NodeName}; retrying next pass",
_nodeName);
_statusWriteFailureLogged = true;
}
}
/// <summary>
/// Handles a tombstoned <c>deployed_secrets</c> row: deletes it from this node's store, but
/// only if this node had previously ledgered it as mirrored — an out-of-band, same-named
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Secrets/SecretMirrorStatusReporter.cs
+95
View File
@@ -0,0 +1,95 @@
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.ScadaBridge.HealthMonitoring;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Secrets;
/// <summary>
/// Review finding M1: site-side hosted service that periodically reads EVERY node's row from
/// the LocalDb-replicated <c>secret_mirror_status</c> table and pushes the list into
/// <see cref="ISiteHealthCollector.SetSecretMirrorStatuses"/>, so the next health report
/// carries each site node's deployed-secrets mirror failures — the standby's included.
/// </summary>
/// <remarks>
/// <para>
/// Runs on every site node (the read is a cheap local SELECT), but only the active node's
/// report is ever sent (<c>HealthReportSender</c>), which is exactly why the status lives in a
/// REPLICATED table: the standby writes its row, CDC carries it to the active node, and this
/// reporter on the active node forwards it. A site pair WITHOUT LocalDb replication reports
/// only the active node's row — the same prerequisite the deployed secrets themselves have.
/// </para>
/// <para>
/// Lives in SiteRuntime (which already references HealthMonitoring and owns
/// <see cref="SiteStorageService"/>), following <c>SiteStreamAlarmDropReporter</c>: immediate
/// first probe, fixed cadence, exceptions logged and swallowed. A failed read leaves the
/// previous snapshot on the collector rather than clearing it.
/// </para>
/// </remarks>
public sealed class SecretMirrorStatusReporter : BackgroundService
{
/// <summary>Default poll cadence (30 s), matching the site health report interval.</summary>
internal static readonly TimeSpan DefaultPollInterval = TimeSpan.FromSeconds(30);
private readonly ISiteHealthCollector _collector;
private readonly SiteStorageService _storage;
private readonly ILogger<SecretMirrorStatusReporter> _logger;
private readonly TimeSpan _pollInterval;
/// <summary>Initializes a new instance of <see cref="SecretMirrorStatusReporter"/>.</summary>
/// <param name="collector">The site health collector receiving the statuses.</param>
/// <param name="storage">Site storage exposing <c>secret_mirror_status</c>.</param>
/// <param name="logger">Logger instance.</param>
/// <param name="pollInterval">Poll interval override; defaults to <see cref="DefaultPollInterval"/> (30 s).</param>
public SecretMirrorStatusReporter(
ISiteHealthCollector collector,
SiteStorageService storage,
ILogger<SecretMirrorStatusReporter> logger,
TimeSpan? pollInterval = null)
{
_collector = collector ?? throw new ArgumentNullException(nameof(collector));
_storage = storage ?? throw new ArgumentNullException(nameof(storage));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
_pollInterval = pollInterval ?? DefaultPollInterval;
}
/// <inheritdoc />
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
await ProbeAsync().ConfigureAwait(false);
while (!stoppingToken.IsCancellationRequested)
{
try
{
await Task.Delay(_pollInterval, stoppingToken).ConfigureAwait(false);
}
catch (OperationCanceledException)
{
break;
}
await ProbeAsync().ConfigureAwait(false);
}
}
/// <summary>
/// Reads every node's mirror status and pushes it onto the collector; never throws.
/// Public so a test can drive one deterministic probe.
/// </summary>
/// <returns>A task that completes when the probe has finished.</returns>
public async Task ProbeAsync()
{
try
{
var statuses = await _storage.GetSecretMirrorStatusesAsync().ConfigureAwait(false);
_collector.SetSecretMirrorStatuses(statuses);
}
catch (Exception ex)
{
// Catch-all is deliberate: health reporting must never take the node down, and
// the loop must survive so the next tick retries.
_logger.LogWarning(ex, "SecretMirrorStatusReporter probe failed; next tick will retry.");
}
}
}
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs
+7
View File
@@ -140,6 +140,13 @@ public static class ServiceCollectionExtensions
services.AddSingleton<DeployedSecretsMirror>();
services.AddHostedService(sp => sp.GetRequiredService<DeployedSecretsMirror>());
// Review finding M1: forward every node's replicated secret_mirror_status row onto the
// site health report, so the ACTIVE node's report carries the STANDBY's mirror failures.
services.AddHostedService(sp => new SecretMirrorStatusReporter(
sp.GetRequiredService<HealthMonitoring.ISiteHealthCollector>(),
sp.GetRequiredService<SiteStorageService>(),
sp.GetRequiredService<ILogger<SecretMirrorStatusReporter>>()));
return services;
}
tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Pages/HealthPageTests.cs
+64
View File
@@ -559,6 +559,70 @@ public class HealthPageTests : BunitContext
StartupReconcilerActive = active,
};
// ----- Review finding M1: per-node deployed-secrets mirror failures ------------------
//
// The standby's failures ride the active node's report; the card must name the node,
// the secret and the message. Null ("no data") and all-healthy nodes render nothing.
[Fact]
public void Renders_SecretMirrorFailures_LabelledByNode()
{
SeedStates(new SiteHealthState
{
SiteId = "site-a",
IsOnline = true,
LastHeartbeatAt = DateTimeOffset.UtcNow,
LatestReport = ReportWithReconciler("site-a", active: true) with
{
SecretMirrorStatuses =
[
new SecretMirrorNodeStatus("node-a", DateTimeOffset.UtcNow, []),
new SecretMirrorNodeStatus(
"node-b",
DateTimeOffset.UtcNow,
[new SecretMirrorFailure("OPC-PWD", "did not decrypt on this node")]),
],
},
});
var cut = RenderHealthPage();
cut.WaitForAssertion(() =>
{
Assert.Contains("data-test=\"secret-mirror-badge\"", cut.Markup);
var panel = cut.Find("[data-test=secret-mirror-failures]");
Assert.Contains("node-b", panel.TextContent);
Assert.Contains("OPC-PWD", panel.TextContent);
Assert.Contains("did not decrypt on this node", panel.TextContent);
Assert.DoesNotContain("node-a", panel.TextContent);
});
}
[Theory]
[InlineData(false)] // null = "no data" (central self-report, pre-M1 site)
[InlineData(true)] // every node healthy
public void DoesNotRender_SecretMirrorWarning_WhenNoNodeIsFailing(bool reported)
{
SeedStates(new SiteHealthState
{
SiteId = "site-a",
IsOnline = true,
LastHeartbeatAt = DateTimeOffset.UtcNow,
LatestReport = ReportWithReconciler("site-a", active: true) with
{
SecretMirrorStatuses = reported
? [new SecretMirrorNodeStatus("node-a", DateTimeOffset.UtcNow, [])]
: null,
},
});
var cut = RenderHealthPage();
cut.WaitForAssertion(() => Assert.Contains("site-a", cut.Markup));
Assert.DoesNotContain("data-test=\"secret-mirror-badge\"", cut.Markup);
Assert.DoesNotContain("data-test=\"secret-mirror-failures\"", cut.Markup);
}
[Fact]
public void Renders_OfflineSince_ForOfflineSiteWithStatusChange()
{
tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/CentralControlDtoMapperTests.cs
+28
View File
@@ -369,6 +369,7 @@ public class CentralControlDtoMapperTests
Assert.Null(roundTripped.LocalDbReplicationConnected);
Assert.Null(roundTripped.LocalDbOplogBacklog);
Assert.Null(roundTripped.StartupReconcilerActive);
Assert.Null(roundTripped.SecretMirrorStatuses);
}
[Fact]
@@ -384,10 +385,14 @@ public class CentralControlDtoMapperTests
DataConnectionEndpoints = new Dictionary<string, string>(),
DataConnectionTagQuality = new Dictionary<string, TagQualityCounts>(),
ClusterNodes = [],
SecretMirrorStatuses = [],
};
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
Assert.NotNull(roundTripped.SecretMirrorStatuses);
Assert.Empty(roundTripped.SecretMirrorStatuses);
Assert.NotNull(roundTripped.DataConnectionEndpoints);
Assert.Empty(roundTripped.DataConnectionEndpoints);
Assert.NotNull(roundTripped.DataConnectionTagQuality);
@@ -559,6 +564,18 @@ public class CentralControlDtoMapperTests
DetachedScriptThreads = 2,
ScriptRunShedCount = 13,
SiteStreamAlarmDropCount = 41L,
// Review finding M1: every node's secret mirror status, standby included.
SecretMirrorStatuses =
[
new SecretMirrorNodeStatus("node-a", SampleInstant, []),
new SecretMirrorNodeStatus(
"node-b",
SampleInstant.AddMinutes(-3),
[
new SecretMirrorFailure("OPC-PWD", "did not decrypt on this node"),
new SecretMirrorFailure("MX-KEY", "could not be confirmed applied"),
]),
],
};
/// <summary>
@@ -611,6 +628,17 @@ public class CentralControlDtoMapperTests
Assert.Equal(expected.DetachedScriptThreads, actual.DetachedScriptThreads);
Assert.Equal(expected.ScriptRunShedCount, actual.ScriptRunShedCount);
Assert.Equal(expected.SiteStreamAlarmDropCount, actual.SiteStreamAlarmDropCount);
Assert.Equal(expected.SecretMirrorStatuses is null, actual.SecretMirrorStatuses is null);
if (expected.SecretMirrorStatuses is not null)
{
Assert.Equal(expected.SecretMirrorStatuses.Count, actual.SecretMirrorStatuses!.Count);
for (var i = 0; i < expected.SecretMirrorStatuses.Count; i++)
{
Assert.Equal(expected.SecretMirrorStatuses[i].NodeName, actual.SecretMirrorStatuses[i].NodeName);
Assert.Equal(expected.SecretMirrorStatuses[i].CheckedAtUtc, actual.SecretMirrorStatuses[i].CheckedAtUtc);
Assert.Equal(expected.SecretMirrorStatuses[i].Failures, actual.SecretMirrorStatuses[i].Failures);
}
}
}
/// <summary>
tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/SecretMirrorStatusHealthFieldTests.cs
+45
View File
@@ -0,0 +1,45 @@
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
namespace ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests;
/// <summary>
/// Review finding M1 — <c>SiteHealthReport.SecretMirrorStatuses</c> carries every site node's
/// deployed-secrets mirror failures (the standby's included) on the active node's report.
/// </summary>
public class SecretMirrorStatusHealthFieldTests
{
[Fact]
public void CollectReport_NeverTold_LeavesTheFieldNull()
{
Assert.Null(new SiteHealthCollector().CollectReport("TestSite").SecretMirrorStatuses);
}
[Fact]
public void CollectReport_CarriesTheLatestSnapshot_AndDoesNotDrainIt()
{
var collector = new SiteHealthCollector();
collector.SetSecretMirrorStatuses(
[
new SecretMirrorNodeStatus("node-a", DateTimeOffset.UtcNow, []),
new SecretMirrorNodeStatus(
"node-b", DateTimeOffset.UtcNow, [new SecretMirrorFailure("S1", "did not decrypt")]),
]);
var first = collector.CollectReport("TestSite").SecretMirrorStatuses!;
var second = collector.CollectReport("TestSite").SecretMirrorStatuses!;
Assert.Equal(["node-a", "node-b"], first.Select(s => s.NodeName));
Assert.Equal(2, second.Count); // point-in-time, not interval-drained
Assert.Equal("S1", Assert.Single(first[1].Failures).SecretName);
}
[Fact]
public void SetNull_ResetsToNoData()
{
var collector = new SiteHealthCollector();
collector.SetSecretMirrorStatuses([]);
collector.SetSecretMirrorStatuses(null);
Assert.Null(collector.CollectReport("TestSite").SecretMirrorStatuses);
}
}
tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbCdcRegistrationTests.cs
+3 -3
View File
@@ -212,7 +212,7 @@ public class SiteLocalDbCdcRegistrationTests : IDisposable
// ---- helpers ----------------------------------------------------------------------
/// <summary>
/// The eleven replicated tables, plus the two deliberately-unregistered central-only
/// The twelve replicated tables, plus the two deliberately-unregistered central-only
/// notification tables, plus the node-local (also deliberately-unregistered)
/// <c>mirrored_secrets</c> ledger.
/// </summary>
@@ -221,7 +221,7 @@ public class SiteLocalDbCdcRegistrationTests : IDisposable
"OperationTracking", "site_events", "sf_messages", "deployed_configurations",
"static_attribute_overrides", "shared_scripts", "external_systems",
"database_connections", "data_connection_definitions", "native_alarm_state",
"deployed_secrets",
"deployed_secrets", "secret_mirror_status",
"notification_lists", "smtp_configurations", "mirrored_secrets",
];
@@ -230,7 +230,7 @@ public class SiteLocalDbCdcRegistrationTests : IDisposable
"OperationTracking", "site_events", "sf_messages", "deployed_configurations",
"static_attribute_overrides", "shared_scripts", "external_systems",
"database_connections", "data_connection_definitions", "native_alarm_state",
"deployed_secrets",
"deployed_secrets", "secret_mirror_status",
];
private static void AssertCaptureTriggersCoverTheReplicatedTables(ILocalDb db)
tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbReplicatedTableFkGuardTests.cs
+3 -2
View File
@@ -15,8 +15,9 @@ namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
/// <c>RegisterReplicated</c> time, which on a site node is a BOOT failure of the replicating
/// pair. This test moves that discovery from the first replicated boot to the build: it walks
/// the real site schema (every table <c>SiteLocalDbSetup.OnReady</c> creates) and asserts none
/// of the eleven <see cref="SiteLocalDbSetup.ReplicatedTables"/> (incl. <c>deployed_secrets</c>,
/// added for site-secrets artifact deploy) has such an inbound FK.
/// of the twelve <see cref="SiteLocalDbSetup.ReplicatedTables"/> (incl. <c>deployed_secrets</c>,
/// added for site-secrets artifact deploy, and <c>secret_mirror_status</c>, added by review
/// finding M1) has such an inbound FK.
/// </summary>
/// <remarks>
/// The schema is built with an UNREPLICATED config on purpose: a replicated one would make
tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs
+7 -5
View File
@@ -254,7 +254,7 @@ public class SiteLocalDbWiringTests : IDisposable
"deployed_configurations", "static_attribute_overrides", "shared_scripts",
"external_systems", "database_connections", "notification_lists",
"data_connection_definitions", "smtp_configurations", "native_alarm_state",
"deployed_secrets", "mirrored_secrets",
"deployed_secrets", "mirrored_secrets", "secret_mirror_status",
})
{
Assert.Contains(table, tables);
@@ -262,9 +262,10 @@ public class SiteLocalDbWiringTests : IDisposable
}
[Fact]
public void Site_LocalDb_RegistersExactlyTheElevenReplicatedTables()
public void Site_LocalDb_RegistersExactlyTheTwelveReplicatedTables()
{
// The Phase 2 cutover, extended by site-secrets artifact deploy's deployed_secrets.
// The Phase 2 cutover, extended by site-secrets artifact deploy's deployed_secrets
// and review finding M1's per-node secret_mirror_status.
// This is an EXACTLY check rather than a Contains in both directions, and both
// directions are load-bearing.
//
@@ -285,8 +286,8 @@ public class SiteLocalDbWiringTests : IDisposable
// data_connection_definitions precedes database_connections.
"OperationTracking", "data_connection_definitions", "database_connections",
"deployed_configurations", "deployed_secrets", "external_systems",
"native_alarm_state", "sf_messages", "shared_scripts", "site_events",
"static_attribute_overrides",
"native_alarm_state", "secret_mirror_status", "sf_messages", "shared_scripts",
"site_events", "static_attribute_overrides",
],
db.ReplicatedTables.Keys.OrderBy(k => k, StringComparer.Ordinal).ToArray());
}
@@ -334,6 +335,7 @@ public class SiteLocalDbWiringTests : IDisposable
["instance_unique_name", "source_canonical_name", "source_reference"],
db.ReplicatedTables["native_alarm_state"].PkColumns);
Assert.Equal(["name"], db.ReplicatedTables["deployed_secrets"].PkColumns);
Assert.Equal(["node_name"], db.ReplicatedTables["secret_mirror_status"].PkColumns);
}
private static HashSet<string> TableNames(ILocalDb db)
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/DeployedSecretsStorageTests.cs
+49
View File
@@ -1,6 +1,7 @@
using System.Text.Json;
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
using ZB.MOM.WW.ScadaBridge.TestSupport;
@@ -303,6 +304,54 @@ public class DeployedSecretsStorageTests : IAsyncLifetime, IDisposable
Assert.Equal("A", mirrored.Name);
}
// ── secret_mirror_status (review finding M1) ──
[Fact]
public async Task SecretMirrorStatus_RoundTrips_OneRowPerNode_OrderedByNodeName()
{
var at = new DateTimeOffset(2026, 9, 28, 6, 0, 0, TimeSpan.Zero);
await _storage.SetSecretMirrorStatusAsync(
"node-b", [new SecretMirrorFailure("A", "did not decrypt")], at);
await _storage.SetSecretMirrorStatusAsync("node-a", [], at.AddMinutes(1));
var rows = await _storage.GetSecretMirrorStatusesAsync();
Assert.Equal(["node-a", "node-b"], rows.Select(r => r.NodeName));
Assert.Empty(rows[0].Failures);
Assert.Equal(at.AddMinutes(1), rows[0].CheckedAtUtc);
var failure = Assert.Single(rows[1].Failures);
Assert.Equal(new SecretMirrorFailure("A", "did not decrypt"), failure);
Assert.Equal(at, rows[1].CheckedAtUtc);
}
[Fact]
public async Task SecretMirrorStatus_Set_UpsertsTheNodesRow()
{
var at = new DateTimeOffset(2026, 9, 28, 6, 0, 0, TimeSpan.Zero);
await _storage.SetSecretMirrorStatusAsync("node-a", [new SecretMirrorFailure("A", "x")], at);
await _storage.SetSecretMirrorStatusAsync("node-a", [], at.AddHours(1));
var row = Assert.Single(await _storage.GetSecretMirrorStatusesAsync());
Assert.Empty(row.Failures);
Assert.Equal(at.AddHours(1), row.CheckedAtUtc);
}
[Fact]
public async Task SecretMirrorStatus_UnparseableRow_ReadsAsAFailure_NotAsHealthy()
{
await using (var connection = _storage.CreateConnection())
await using (var insert = connection.CreateCommand())
{
insert.CommandText =
"INSERT INTO secret_mirror_status (node_name, failures_json, checked_at) VALUES ('node-b', '{not json', '2026-09-28T00:00:00.0000000+00:00')";
await insert.ExecuteNonQueryAsync();
}
var row = Assert.Single(await _storage.GetSecretMirrorStatusesAsync());
Assert.Equal("node-b", row.NodeName);
Assert.Single(row.Failures);
}
private static SecretArtifact Art(string name, long revision) => new(
Name: name,
ContentType: "application/octet-stream",
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/SiteStorageSchemaTests.cs
+2 -2
View File
@@ -23,14 +23,14 @@ public class SiteStorageSchemaTests
SiteStorageSchema.Apply(connection);
SiteStorageSchema.Apply(connection); // second run must not throw
// Every table is named explicitly on purpose — a loop asserting "11 tables"
// Every table is named explicitly on purpose — a loop asserting "12 tables"
// would still pass if one were renamed.
foreach (var table in new[]
{
"deployed_configurations", "static_attribute_overrides", "shared_scripts",
"external_systems", "database_connections", "notification_lists",
"data_connection_definitions", "smtp_configurations", "native_alarm_state",
"deployed_secrets", "mirrored_secrets",
"deployed_secrets", "mirrored_secrets", "secret_mirror_status",
})
{
Assert.True(TableExists(connection, table), $"missing table: {table}");
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Secrets/DeployedSecretsMirrorTests.cs
+131 -2
View File
@@ -2,6 +2,7 @@ using System.Collections.Concurrent;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Secrets;
@@ -655,6 +656,132 @@ public class DeployedSecretsMirrorTests : IDisposable
}
}
// ── Per-node status row for central (review finding M1) ──
[Fact]
public async Task Status_FirstPassAfterStartup_WritesThisNodesRow_EvenWhenHealthy()
{
var time = new FakeTimeProvider(new DateTimeOffset(2026, 9, 28, 6, 0, 0, TimeSpan.Zero));
var (mirror, storage, _, _, _) = await CreateAsync(time: time, nodeIdentity: new FixedNodeIdentity("node-b"));
await mirror.ReconcileAsync(CancellationToken.None);
var status = Assert.Single(await storage.GetSecretMirrorStatusesAsync());
Assert.Equal("node-b", status.NodeName);
Assert.Empty(status.Failures);
Assert.Equal(time.Now, status.CheckedAtUtc);
}
[Fact]
public async Task Status_UnchangedFailureSet_IsNotRewrittenOnLaterPasses()
{
var time = new FakeTimeProvider(new DateTimeOffset(2026, 9, 28, 6, 0, 0, TimeSpan.Zero));
var (mirror, storage, _, resolver, _) = await CreateAsync(time: time, nodeIdentity: new FixedNodeIdentity("node-b"));
resolver.ThrowFor("a");
await storage.UpsertDeployedSecretsAsync([Art("a", 1, kek: "kek-central")], "d1");
await mirror.ReconcileAsync(CancellationToken.None);
var firstWrite = Assert.Single(await storage.GetSecretMirrorStatusesAsync());
// Same failure on every later pass: the row is replicated, so it must not be rewritten.
time.Now = time.Now.AddMinutes(5);
await mirror.ReconcileAsync(CancellationToken.None);
await mirror.ReconcileAsync(CancellationToken.None);
var after = Assert.Single(await storage.GetSecretMirrorStatusesAsync());
Assert.Equal(firstWrite.CheckedAtUtc, after.CheckedAtUtc);
var failure = Assert.Single(after.Failures);
Assert.Equal("a", failure.SecretName);
Assert.Contains("kek-central", failure.Message);
}
[Fact]
public async Task Status_FailureSetChange_RewritesTheRow_AndClearsWhenHealed()
{
var time = new FakeTimeProvider(new DateTimeOffset(2026, 9, 28, 6, 0, 0, TimeSpan.Zero));
var (mirror, storage, _, resolver, _) = await CreateAsync(time: time, nodeIdentity: new FixedNodeIdentity("node-a"));
await storage.UpsertDeployedSecretsAsync([Art("a", 1)], "d1");
await mirror.ReconcileAsync(CancellationToken.None);
Assert.Empty(Assert.Single(await storage.GetSecretMirrorStatusesAsync()).Failures);
// A decrypt failure appears -> row rewritten with the failure.
resolver.ThrowFor("a");
time.Now = time.Now.AddMinutes(1);
await mirror.ReconcileAsync(CancellationToken.None);
var failing = Assert.Single(await storage.GetSecretMirrorStatusesAsync());
Assert.Equal("a", Assert.Single(failing.Failures).SecretName);
Assert.Equal(time.Now, failing.CheckedAtUtc);
// Healed -> row rewritten empty.
resolver.ClearFailures();
time.Now = time.Now.AddMinutes(1);
await mirror.ReconcileAsync(CancellationToken.None);
var healed = Assert.Single(await storage.GetSecretMirrorStatusesAsync());
Assert.Empty(healed.Failures);
Assert.Equal(time.Now, healed.CheckedAtUtc);
}
[Fact]
public async Task Status_RowCarriesNamesAndMessagesOnly_NeverCiphertext()
{
var (mirror, storage, _, resolver, _) = await CreateAsync(nodeIdentity: new FixedNodeIdentity("node-a"));
resolver.ThrowFor("a");
await storage.UpsertDeployedSecretsAsync([Art("a", 1)], "d1");
await mirror.ReconcileAsync(CancellationToken.None);
await using var connection = storage.CreateConnection();
await using var command = connection.CreateCommand();
command.CommandText = "SELECT failures_json FROM secret_mirror_status WHERE node_name = 'node-a'";
var json = (string)(await command.ExecuteScalarAsync())!;
Assert.Contains("\"SecretName\":\"a\"", json);
Assert.DoesNotContain("Ciphertext", json);
Assert.DoesNotContain(Convert.ToBase64String(new byte[] { 1, 2, 3, 4, 5 }), json);
}
[Fact]
public async Task Status_WriteFailure_DoesNotBreakThePass_AndIsRetriedNextPass()
{
var logger = new CapturingLogger<DeployedSecretsMirror>();
var (mirror, storage, store, _, _) = await CreateAsync(logger: logger, nodeIdentity: new FixedNodeIdentity("node-a"));
await storage.UpsertDeployedSecretsAsync([Art("a", 1)], "d1");
await using (var connection = storage.CreateConnection())
await using (var drop = connection.CreateCommand())
{
drop.CommandText = "DROP TABLE secret_mirror_status";
await drop.ExecuteNonQueryAsync();
}
var r = await mirror.ReconcileAsync(CancellationToken.None);
Assert.Equal(["a"], r.Applied);
Assert.True(store.Contains("a"));
Assert.Contains(logger.Entries, e => e.Level == LogLevel.Warning && e.Message.Contains("mirror status"));
// Table back (e.g. transient fault cleared): the unchanged failure set is still
// written, because the earlier write never landed.
await storage.InitializeAsync();
await mirror.ReconcileAsync(CancellationToken.None);
Assert.Equal("node-a", Assert.Single(await storage.GetSecretMirrorStatusesAsync()).NodeName);
}
[Fact]
public async Task Status_WithoutNodeIdentity_FallsBackToMachineName()
{
var (mirror, storage, _, _, _) = await CreateAsync();
await mirror.ReconcileAsync(CancellationToken.None);
Assert.Equal(Environment.MachineName, Assert.Single(await storage.GetSecretMirrorStatusesAsync()).NodeName);
}
private sealed class FixedNodeIdentity(string? name) : INodeIdentityProvider
{
public string? NodeName { get; } = name;
}
// ── Harness ──
private async Task<(
@@ -665,7 +792,8 @@ public class DeployedSecretsMirrorTests : IDisposable
FakeSecretCacheInvalidator Invalidator)> CreateAsync(
CapturingLogger<DeployedSecretsMirror>? logger = null,
TimeSpan? interval = null,
TimeProvider? time = null)
TimeProvider? time = null,
INodeIdentityProvider? nodeIdentity = null)
{
var storage = new SiteStorageService(_localDb.Db, NullLogger<SiteStorageService>.Instance);
await storage.InitializeAsync();
@@ -685,7 +813,8 @@ public class DeployedSecretsMirrorTests : IDisposable
invalidator,
options,
(ILogger<DeployedSecretsMirror>?)logger ?? NullLogger<DeployedSecretsMirror>.Instance,
time ?? TimeProvider.System);
time ?? TimeProvider.System,
nodeIdentity);
return (mirror, storage, store, resolver, invalidator);
}
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Secrets/SecretMirrorStatusReporterTests.cs
+95
View File
@@ -0,0 +1,95 @@
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
using ZB.MOM.WW.ScadaBridge.HealthMonitoring;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Secrets;
using ZB.MOM.WW.ScadaBridge.TestSupport;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Secrets;
/// <summary>
/// Review finding M1: <see cref="SecretMirrorStatusReporter"/> forwards every node's
/// replicated <c>secret_mirror_status</c> row onto the site health report via the collector.
/// </summary>
public class SecretMirrorStatusReporterTests : IAsyncLifetime, IDisposable
{
private readonly TestLocalDb _localDb = TestLocalDb.CreateTemp("secret-mirror-status");
private SiteStorageService _storage = null!;
public async Task InitializeAsync()
{
_storage = new SiteStorageService(_localDb.Db, NullLogger<SiteStorageService>.Instance);
await _storage.InitializeAsync();
}
public Task DisposeAsync() => Task.CompletedTask;
public void Dispose()
{
var path = _localDb.Path;
_localDb.Dispose();
TestLocalDb.DeleteFiles(path);
}
[Fact]
public async Task Probe_PushesEveryNodesRow_OntoTheNextReport()
{
var at = new DateTimeOffset(2026, 9, 28, 6, 0, 0, TimeSpan.Zero);
await _storage.SetSecretMirrorStatusAsync("node-a", [], at);
await _storage.SetSecretMirrorStatusAsync(
"node-b", [new SecretMirrorFailure("OPC-PWD", "did not decrypt on this node")], at);
var collector = new SiteHealthCollector();
var reporter = new SecretMirrorStatusReporter(
collector, _storage, NullLogger<SecretMirrorStatusReporter>.Instance);
await reporter.ProbeAsync();
var report = collector.CollectReport("site-a");
Assert.NotNull(report.SecretMirrorStatuses);
Assert.Equal(["node-a", "node-b"], report.SecretMirrorStatuses!.Select(s => s.NodeName));
Assert.Equal("OPC-PWD", Assert.Single(report.SecretMirrorStatuses[1].Failures).SecretName);
}
[Fact]
public void BeforeAnyProbe_TheReportFieldIsNull()
{
var report = new SiteHealthCollector().CollectReport("site-a");
Assert.Null(report.SecretMirrorStatuses);
}
[Fact]
public async Task Probe_NoRowsYet_PushesAnEmptyList_NotNull()
{
var collector = new SiteHealthCollector();
var reporter = new SecretMirrorStatusReporter(
collector, _storage, NullLogger<SecretMirrorStatusReporter>.Instance);
await reporter.ProbeAsync();
Assert.Empty(collector.CollectReport("site-a").SecretMirrorStatuses!);
}
[Fact]
public async Task Probe_ReadFailure_KeepsThePreviousSnapshot_AndDoesNotThrow()
{
await _storage.SetSecretMirrorStatusAsync("node-a", [], DateTimeOffset.UtcNow);
var collector = new SiteHealthCollector();
var reporter = new SecretMirrorStatusReporter(
collector, _storage, NullLogger<SecretMirrorStatusReporter>.Instance);
await reporter.ProbeAsync();
await using (var connection = _storage.CreateConnection())
await using (var drop = connection.CreateCommand())
{
drop.CommandText = "DROP TABLE secret_mirror_status";
await drop.ExecuteNonQueryAsync();
}
await reporter.ProbeAsync();
// Point-in-time: not drained on collect, and not cleared by a failed read.
Assert.Single(collector.CollectReport("site-a").SecretMirrorStatuses!);
Assert.Single(collector.CollectReport("site-a").SecretMirrorStatuses!);
}
}