fix(site-secrets): prune secret_mirror_status rows of decommissioned or renamed nodes
Each node now re-writes its mirror-status row every SecretMirrorStatusRefreshInterval (10 min) as a heartbeat; SecretMirrorStatusReporter leaves off the health report and deletes any row not refreshed for SecretMirrorStatusStaleAfter (1 h, validated >= 2x the refresh). The delete is conditional on the checked_at it read, so a node that came back keeps its fresh row; a returning node re-writes its row on its first pass.
This commit is contained in:
@@ -141,7 +141,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
|
|||||||
- Per-instance operation lock covers all mutating commands (deploy, disable, enable, delete).
|
- Per-instance operation lock covers all mutating commands (deploy, disable, enable, delete).
|
||||||
- **Bulk site deployment (`DeploymentService.DeploySiteAsync`, arch-review WP2.5)** redeploys every out-of-date instance at a site through one flatten session (a template chain shared by N instances is walked once) in three phases — serial Prepare, bounded-parallel Send (`SiteDeploymentMaxParallelism`, default 4), serial Finalize — still **not all-or-nothing**: any one instance's failure doesn't block the rest. **Disabled instances are SKIPPED by bulk deploy as of 2026-08-19 (arch-review P7-2)** — they are reported in the summary's additive `SkippedInstances` list, never flattened/sent/re-enabled (single-instance deploy remains the explicit operator path that deploys+enables a Disabled instance); **Up-to-date instances are ALSO skipped as of 2026-08-20 (arch-review P7-6)** — a redeploy is not idempotent at the site (it resets static overrides and clears native alarm state), so `DeploySiteAsync` pre-filters through `IStaleInstanceProbe` (trustworthy since P7-4 fixed its memo timing) and reports unchanged instances in `SkippedInstances` with reason `up-to-date`; the filter is conservative (every uncertainty deploys: missing snapshot, `ShouldQuerySiteBeforeRedeploy`, null/faulting probe). Additive `MgmtDeploySiteCommand.Force` / CLI `deploy site --force` restores the unconditional push for what central cannot see (restored volume, rebuilt node) — **Disabled still wins under `--force`**. Also P7-5: the three phases share one scoped DbContext and a failed `SaveChanges` leaves the entity tracked, so one permanent failure used to cascade to every later instance and strand records `InProgress`; `MarkDeploymentFailedAsync` now detaches pending changes and retries the `Failed` write once on the clean tracker; Prepare's per-instance reconcile also runs under a short `SiteDeploymentReconcileTimeout` (5s) with a once-per-batch unreachable-site circuit-breaker, so a down site no longer costs 30s × N serial before the first send (P7-3). The `PendingDeployment` staging row (5-minute TTL) is written per-instance immediately before that instance's round-trip in the Send phase, not upfront in Prepare, so a large batch's tail instances don't have their TTL clock already half-spent by the time they're reached.
|
- **Bulk site deployment (`DeploymentService.DeploySiteAsync`, arch-review WP2.5)** redeploys every out-of-date instance at a site through one flatten session (a template chain shared by N instances is walked once) in three phases — serial Prepare, bounded-parallel Send (`SiteDeploymentMaxParallelism`, default 4), serial Finalize — still **not all-or-nothing**: any one instance's failure doesn't block the rest. **Disabled instances are SKIPPED by bulk deploy as of 2026-08-19 (arch-review P7-2)** — they are reported in the summary's additive `SkippedInstances` list, never flattened/sent/re-enabled (single-instance deploy remains the explicit operator path that deploys+enables a Disabled instance); **Up-to-date instances are ALSO skipped as of 2026-08-20 (arch-review P7-6)** — a redeploy is not idempotent at the site (it resets static overrides and clears native alarm state), so `DeploySiteAsync` pre-filters through `IStaleInstanceProbe` (trustworthy since P7-4 fixed its memo timing) and reports unchanged instances in `SkippedInstances` with reason `up-to-date`; the filter is conservative (every uncertainty deploys: missing snapshot, `ShouldQuerySiteBeforeRedeploy`, null/faulting probe). Additive `MgmtDeploySiteCommand.Force` / CLI `deploy site --force` restores the unconditional push for what central cannot see (restored volume, rebuilt node) — **Disabled still wins under `--force`**. Also P7-5: the three phases share one scoped DbContext and a failed `SaveChanges` leaves the entity tracked, so one permanent failure used to cascade to every later instance and strand records `InProgress`; `MarkDeploymentFailedAsync` now detaches pending changes and retries the `Failed` write once on the clean tracker; Prepare's per-instance reconcile also runs under a short `SiteDeploymentReconcileTimeout` (5s) with a once-per-batch unreachable-site circuit-breaker, so a down site no longer costs 30s × N serial before the first send (P7-3). The `PendingDeployment` staging row (5-minute TTL) is written per-instance immediately before that instance's round-trip in the Send phase, not upfront in Prepare, so a large batch's tail instances don't have their TTL clock already half-spent by the time they're reached.
|
||||||
- **Site artifact revision + fleet-wide pending-updates indicator (2026-09-14, `docs/plans/2026-09-14-pending-updates-design.md`).** `Site.ArtifactRevisionHash`/`ArtifactsDeployedAt` (migration `AddSiteArtifactRevision`) are stamped by `ArtifactDeploymentService` only on a site whose artifact deploy succeeds, so a redeployed fleet's artifacts are trackable per site the same way instance staleness already is. `PendingUpdatesEvaluator` combines that with `DeploymentService.GetInstanceStalenessAsync` (one shared `FlattenSession` for the whole fleet) into a `PendingUpdatesSnapshot`, held behind a process-wide singleton `IPendingUpdatesCache` (`ScadaBridge:DeploymentManager:PendingUpdatesCacheTtl`, default 60s) so N Blazor circuits cost one fleet flatten per TTL, not one each. Surfaces: the Sites page's per-site **Artifacts** badge (`Current`/`Out of date`/`Never deployed`) + instance-staleness line + header **Re-check**, and a rail-footer **pending updates** indicator (Deployer/Administrator only, 30s poll, site-scoped). **Upgrade consequence:** every site reads *Never deployed* until one artifact deploy runs post-upgrade — deliberate, not a bug. Also shipped: the rail nav now sits in its own scroll pane (`NavMenu`'s `.rail-scroll` + `site.css`) so a long nav can't push the footer/indicator off-screen; the `ZB.MOM.WW.Theme` kit itself is untouched (register row 123 proposes upstreaming it).
|
- **Site artifact revision + fleet-wide pending-updates indicator (2026-09-14, `docs/plans/2026-09-14-pending-updates-design.md`).** `Site.ArtifactRevisionHash`/`ArtifactsDeployedAt` (migration `AddSiteArtifactRevision`) are stamped by `ArtifactDeploymentService` only on a site whose artifact deploy succeeds, so a redeployed fleet's artifacts are trackable per site the same way instance staleness already is. `PendingUpdatesEvaluator` combines that with `DeploymentService.GetInstanceStalenessAsync` (one shared `FlattenSession` for the whole fleet) into a `PendingUpdatesSnapshot`, held behind a process-wide singleton `IPendingUpdatesCache` (`ScadaBridge:DeploymentManager:PendingUpdatesCacheTtl`, default 60s) so N Blazor circuits cost one fleet flatten per TTL, not one each. Surfaces: the Sites page's per-site **Artifacts** badge (`Current`/`Out of date`/`Never deployed`) + instance-staleness line + header **Re-check**, and a rail-footer **pending updates** indicator (Deployer/Administrator only, 30s poll, site-scoped). **Upgrade consequence:** every site reads *Never deployed* until one artifact deploy runs post-upgrade — deliberate, not a bug. Also shipped: the rail nav now sits in its own scroll pane (`NavMenu`'s `.rail-scroll` + `site.css`) so a long nav can't push the footer/indicator off-screen; the `ZB.MOM.WW.Theme` kit itself is untouched (register row 123 proposes upstreaming it).
|
||||||
- **Site secrets ride the artifact deploy; the gRPC secrets hub is retired in ScadaBridge (2026-09-28, `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`).** `Deploy Artifacts` now ships each site **only the secrets it references** — `SiteSecretsCollector` scans a site's data-connection JSON for `secret:<name>` (`SecretReferenceScanner`, protocol-agnostic, bootstrap names like `SB-GRPC-PSK-*` excluded) and resolves each against the central `ISecretStore`; a missing/tombstoned secret fails **that site's** deploy before anything is sent (fail-closed), ciphertext only ever crosses the wire, and secrets factor into `ArtifactRevisionHashService`'s staleness hash (a missing secret hashes as a distinguishable marker, so an unresolvable reference never reads *Current*; a site with no secret references hashes identically whether the field is null or empty). A pre-feature site (answers with no `SecretsApplied`) or a partial apply is also treated as a deploy failure and is never stamped; one `DeploySiteSecrets`/`SiteSecrets` audit row (`secrets` name + revision, `missing` and `removed` names, never ciphertext — `removed` comes from the site's `SecretsRemoved` reply, so a removal-only deploy is audited too) is written per site deploy that carried, lacked or retired secrets, on both the sweep and the per-site retry paths. A central secret-store read failure fails only the sites that reference a secret and leaves them *Out of date* in the pending-updates check — it no longer aborts the sweep or the fleet-wide check — and both artifact-deploy CLI commands exit 1 when any site failed. Site-side: `deployed_secrets` is the 11th LocalDb-replicated table (ciphertext rows only, upsert-then-soft-remove); the per-node `DeployedSecretsMirror` (`SiteRuntime:DeployedSecretsMirrorInterval`, default 15s, also run synchronously right after an active-node apply) reconciles it into that node's local `ISecretStore`, ledgers what it mirrored in the node-local `mirrored_secrets` table so an out-of-band local secret sharing a deployed name is never silently touched (a mirror-owned name deleted or edited locally is healed back to the deployed value on the next pass, even at an unchanged revision), and trial-decrypts every pass to catch a KEK mismatch. Each node also records its current mirror failures (names + messages only) in the 12th replicated table `secret_mirror_status` — written only when its failure set changes, plus once per start-up — and the active node forwards every node's row as `SiteHealthReport.SecretMirrorStatuses` (proto field 33), so a **standby** KEK mismatch shows as a *Secret mirror failing* badge on the Health dashboard instead of only in the standby's log (review finding M1). Secrets are mirrored before the deploy's data connections reach the DCL; a rotation with unchanged connection config takes effect on the next (re)connect. **A site pair using deployed secrets must run LocalDb replication** — otherwise only the node active at deploy time holds them. Central runs one shared secret store, `Secrets:Store:Provider=SqlServer` (Central role only; a clustered central left on `Sqlite` gets a boot Warning) — the pre-host `${secret:}` expander reads the same store the running node serves. **ALL secrets replication is retired in ScadaBridge** — the pull-only gRPC hub and the SQL-Server replication mode both — `Secrets:Replication:Enabled` set to anything but empty/`false` now fails `StartupValidator` on either role with a message pointing at Deploy Artifacts / `Secrets:Store:Provider=SqlServer`.
|
- **Site secrets ride the artifact deploy; the gRPC secrets hub is retired in ScadaBridge (2026-09-28, `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`).** `Deploy Artifacts` now ships each site **only the secrets it references** — `SiteSecretsCollector` scans a site's data-connection JSON for `secret:<name>` (`SecretReferenceScanner`, protocol-agnostic, bootstrap names like `SB-GRPC-PSK-*` excluded) and resolves each against the central `ISecretStore`; a missing/tombstoned secret fails **that site's** deploy before anything is sent (fail-closed), ciphertext only ever crosses the wire, and secrets factor into `ArtifactRevisionHashService`'s staleness hash (a missing secret hashes as a distinguishable marker, so an unresolvable reference never reads *Current*; a site with no secret references hashes identically whether the field is null or empty). A pre-feature site (answers with no `SecretsApplied`) or a partial apply is also treated as a deploy failure and is never stamped; one `DeploySiteSecrets`/`SiteSecrets` audit row (`secrets` name + revision, `missing` and `removed` names, never ciphertext — `removed` comes from the site's `SecretsRemoved` reply, so a removal-only deploy is audited too) is written per site deploy that carried, lacked or retired secrets, on both the sweep and the per-site retry paths. A central secret-store read failure fails only the sites that reference a secret and leaves them *Out of date* in the pending-updates check — it no longer aborts the sweep or the fleet-wide check — and both artifact-deploy CLI commands exit 1 when any site failed. Site-side: `deployed_secrets` is the 11th LocalDb-replicated table (ciphertext rows only, upsert-then-soft-remove); the per-node `DeployedSecretsMirror` (`SiteRuntime:DeployedSecretsMirrorInterval`, default 15s, also run synchronously right after an active-node apply) reconciles it into that node's local `ISecretStore`, ledgers what it mirrored in the node-local `mirrored_secrets` table so an out-of-band local secret sharing a deployed name is never silently touched (a mirror-owned name deleted or edited locally is healed back to the deployed value on the next pass, even at an unchanged revision), and trial-decrypts every pass to catch a KEK mismatch. Each node also records its current mirror failures (names + messages only) in the 12th replicated table `secret_mirror_status` — written when its failure set changes, once per start-up and every `SecretMirrorStatusRefreshInterval` (10 min) as a heartbeat; rows not refreshed for `SecretMirrorStatusStaleAfter` (1 h: a decommissioned/renamed/down node) are dropped from the report and deleted — and the active node forwards every node's row as `SiteHealthReport.SecretMirrorStatuses` (proto field 33), so a **standby** KEK mismatch shows as a *Secret mirror failing* badge on the Health dashboard instead of only in the standby's log (review finding M1). Secrets are mirrored before the deploy's data connections reach the DCL; a rotation with unchanged connection config takes effect on the next (re)connect. **A site pair using deployed secrets must run LocalDb replication** — otherwise only the node active at deploy time holds them. Central runs one shared secret store, `Secrets:Store:Provider=SqlServer` (Central role only; a clustered central left on `Sqlite` gets a boot Warning) — the pre-host `${secret:}` expander reads the same store the running node serves. **ALL secrets replication is retired in ScadaBridge** — the pull-only gRPC hub and the SQL-Server replication mode both — `Secrets:Replication:Enabled` set to anything but empty/`false` now fails `StartupValidator` on either role with a message pointing at Deploy Artifacts / `Secrets:Store:Provider=SqlServer`.
|
||||||
- **Create Instance picks its template from the Templates folder tree** (2026-08-21): the flat `<select>` is gone. The Template field is a read-only folder-qualified path box (`TemplatePathFormatter`, "Folder / Sub / Name") + a **Choose…** button, both opening `TemplatePickerDialog` — an `IDialogService.ShowAsync` body wrapping the **same shared `TemplateFolderTree`** the Templates page renders, with its own search, expand/collapse-all, templates-only selection and double-click-to-pick. It opens on the current pick via the tree's additive **`RevealKeys`** parameter (ancestors of the named `f:`/`t:` keys start expanded when no `Filter` is active; an active filter's reveal set still wins; callers passing nothing are byte-identical). `InstanceCreate` gained `[SupplyParameterFromQuery] TemplateId`, which the Templates tree's new **Deployer-gated "New Instance…"** context-menu item navigates with; combined with `?siteId=` it populates the Connection Bindings card on first render. **The trap for anyone extending the picker:** in `TreeViewSelectionMode.Single` the inner `TreeView` reads the *singular* `SelectedKey`/`SelectedKeyChanged` pair, but `TemplateFolderTree` forwards only the plural (Checkbox-mode) `SelectedKeys` pair — so a Single-mode caller cannot get a row highlight through the shared tree and must own selection in its own `NodeContent`, as the picker does. Design: `docs/plans/2026-08-21-template-picker-design.md`.
|
- **Create Instance picks its template from the Templates folder tree** (2026-08-21): the flat `<select>` is gone. The Template field is a read-only folder-qualified path box (`TemplatePathFormatter`, "Folder / Sub / Name") + a **Choose…** button, both opening `TemplatePickerDialog` — an `IDialogService.ShowAsync` body wrapping the **same shared `TemplateFolderTree`** the Templates page renders, with its own search, expand/collapse-all, templates-only selection and double-click-to-pick. It opens on the current pick via the tree's additive **`RevealKeys`** parameter (ancestors of the named `f:`/`t:` keys start expanded when no `Filter` is active; an active filter's reveal set still wins; callers passing nothing are byte-identical). `InstanceCreate` gained `[SupplyParameterFromQuery] TemplateId`, which the Templates tree's new **Deployer-gated "New Instance…"** context-menu item navigates with; combined with `?siteId=` it populates the Connection Bindings card on first render. **The trap for anyone extending the picker:** in `TreeViewSelectionMode.Single` the inner `TreeView` reads the *singular* `SelectedKey`/`SelectedKeyChanged` pair, but `TemplateFolderTree` forwards only the plural (Checkbox-mode) `SelectedKeys` pair — so a Single-mode caller cannot get a row highlight through the shared tree and must own selection in its own `NodeContent`, as the picker does. Design: `docs/plans/2026-08-21-template-picker-design.md`.
|
||||||
- **Create Instance binds module groups at creation** (2026-08-21): the create page's Connection Bindings card flattens template+site through the additive `IFlatteningPipeline.PreviewAsync` (an unsaved instance — no validation, no hash, nothing persisted, instance repo never touched) and reuses `BindingGroupBuilder`; `NodeBrowserDialog.SelectableNodeClasses` lets it pick **Object** nodes (default stays Variable-only for every existing caller). Create = `CreateInstanceAsync` → `SetConnectionBindingsAsync` (the existing upsert, no new server surface) → always navigate to Configure with `?created=1&bound=N|&bindError=…`, because the row exists whether or not the second call succeeded. Partial binding is legal: a group missing a connection or an Object is skipped and stays unbound. Design: `docs/plans/2026-08-21-create-instance-bindings-design.md`.
|
- **Create Instance binds module groups at creation** (2026-08-21): the create page's Connection Bindings card flattens template+site through the additive `IFlatteningPipeline.PreviewAsync` (an unsaved instance — no validation, no hash, nothing persisted, instance repo never touched) and reuses `BindingGroupBuilder`; `NodeBrowserDialog.SelectableNodeClasses` lets it pick **Object** nodes (default stays Variable-only for every existing caller). Create = `CreateInstanceAsync` → `SetConnectionBindingsAsync` (the existing upsert, no new server surface) → always navigate to Configure with `?created=1&bound=N|&bindError=…`, because the row exists whether or not the second call succeeded. Partial binding is legal: a group missing a connection or an Object is skipped and stays unbound. Design: `docs/plans/2026-08-21-create-instance-bindings-design.md`.
|
||||||
- Site-side apply is all-or-nothing per instance.
|
- Site-side apply is all-or-nothing per instance.
|
||||||
|
|||||||
@@ -271,6 +271,10 @@ accurate as the spec:
|
|||||||
`SiteHealthReport.SecretMirrorStatuses` (proto `SiteHealthReportDto` field 33, presence-wrapped),
|
`SiteHealthReport.SecretMirrorStatuses` (proto `SiteHealthReportDto` field 33, presence-wrapped),
|
||||||
and the Central UI health card shows a *Secret mirror failing* badge + per-node list. Requires
|
and the Central UI health card shows a *Secret mirror failing* badge + per-node list. Requires
|
||||||
LocalDb replication for the standby's row to be visible, the same prerequisite as the secrets.
|
LocalDb replication for the standby's row to be visible, the same prerequisite as the secrets.
|
||||||
|
**Stale rows (2026-09-29):** each node also re-writes its row every
|
||||||
|
`SiteRuntime:SecretMirrorStatusRefreshInterval` (10 min) as a heartbeat, and the reporter drops
|
||||||
|
and deletes rows not refreshed for `SiteRuntime:SecretMirrorStatusStaleAfter` (1 h), so a
|
||||||
|
decommissioned or renamed node no longer leaves its row on the Health page.
|
||||||
- **Rig posture.** `docker/docker-compose.yml`'s central nodes now carry
|
- **Rig posture.** `docker/docker-compose.yml`'s central nodes now carry
|
||||||
`Secrets__Store__Provider=SqlServer` plus a shared KEK; site-a shares that KEK (it is the
|
`Secrets__Store__Provider=SqlServer` plus a shared KEK; site-a shares that KEK (it is the
|
||||||
secrets-via-artifact-deploy site); site-b/site-c deliberately carry a **mismatched** KEK so the
|
secrets-via-artifact-deploy site); site-b/site-c deliberately carry a **mismatched** KEK so the
|
||||||
|
|||||||
@@ -39,7 +39,7 @@ Site clusters (metric collection and reporting). Central cluster (aggregation an
|
|||||||
| `AuditRedactionFailure` | Audit Log (central) | Count of payload redactor errors (over-redacted payloads, safety-net hit) since the last interval. |
|
| `AuditRedactionFailure` | Audit Log (central) | Count of payload redactor errors (over-redacted payloads, safety-net hit) since the last interval. |
|
||||||
| `LocalDbReplicationConnected` | Consolidated site LocalDb | Whether a peer sync session is currently running on this node. **Nullable — null means "no reading", not "disconnected".** Reported only by a node with replication configured; an unconfigured node reports **null**, never `false` (register row 80 — see *Unconfigured replication reports null* below). |
|
| `LocalDbReplicationConnected` | Consolidated site LocalDb | Whether a peer sync session is currently running on this node. **Nullable — null means "no reading", not "disconnected".** Reported only by a node with replication configured; an unconfigured node reports **null**, never `false` (register row 80 — see *Unconfigured replication reports null* below). |
|
||||||
| `StartupReconcilerActive` | Host (site) | Whether this node spawned its `SiteReconciliationActor` at boot. **Tri-state:** null = not reported (the central self-report, or a pre-row-73 site build), `true` = running, `false` = the node booted with no `IDeploymentConfigFetcher` and therefore has **no startup self-heal**. Latched once at boot by the Host — not polled. Drives the *No self-heal* dashboard badge (register row 73). |
|
| `StartupReconcilerActive` | Host (site) | Whether this node spawned its `SiteReconciliationActor` at boot. **Tri-state:** null = not reported (the central self-report, or a pre-row-73 site build), `true` = running, `false` = the node booted with no `IDeploymentConfigFetcher` and therefore has **no startup self-heal**. Latched once at boot by the Host — not polled. Drives the *No self-heal* dashboard badge (register row 73). |
|
||||||
| `SecretMirrorStatuses` | Site Runtime (`DeployedSecretsMirror` via the replicated `secret_mirror_status` table) | Every site node's deployed-secrets mirror status — node name, when its failure set last changed, and its current failures (`{SecretName, Message}`; names and messages only, never values). Review finding M1: the **standby's** failures ride the **active** node's report, because each node writes its own row to a LocalDb-replicated table and the active node's `SecretMirrorStatusReporter` reads them all. **Nullable — null = no data** (central self-report, a pre-M1 site, reporter not yet run); an empty list = wired, no row yet; a node with an empty `Failures` list is healthy. Point-in-time (not drained on collect). Wire: `SiteHealthReportDto.secret_mirror_statuses` (field **33**, a presence wrapper so null ≠ empty). Drives the *Secret mirror failing* dashboard badge + per-node failure list. On an unreplicated pair only the active node's own row appears. |
|
| `SecretMirrorStatuses` | Site Runtime (`DeployedSecretsMirror` via the replicated `secret_mirror_status` table) | Every site node's deployed-secrets mirror status — node name, when the node last wrote the row (on a failure-set change, at start-up, or on the 10-min refresh), and its current failures (`{SecretName, Message}`; names and messages only, never values). Review finding M1: the **standby's** failures ride the **active** node's report, because each node writes its own row to a LocalDb-replicated table and the active node's `SecretMirrorStatusReporter` reads them all. **Nullable — null = no data** (central self-report, a pre-M1 site, reporter not yet run); an empty list = wired, no row yet; a node with an empty `Failures` list is healthy. Point-in-time (not drained on collect). Wire: `SiteHealthReportDto.secret_mirror_statuses` (field **33**, a presence wrapper so null ≠ empty). Drives the *Secret mirror failing* dashboard badge + per-node failure list. On an unreplicated pair only the active node's own row appears. A node that has not refreshed its row for `SiteRuntime:SecretMirrorStatusStaleAfter` (default 1 h — decommissioned, renamed or down) is left off and its row deleted. |
|
||||||
| `LocalDbOplogBacklog` | Consolidated site LocalDb | Unacked oplog depth (Prometheus `localdb_oplog_depth`). **Nullable — null means "no reading", NOT "connected with an empty backlog"** (a failed poll rendered as 0 would report a pair that cannot read its own oplog as perfectly healthy). **Expected transient spikes:** the daily `site_events` retention purge and the storage-cap trim are CDC-captured like any other write, so the backlog jumps by the batch size at purge time and drains as the peer acks it — see `Component-SiteEventLogging.md` → Storage and `docs/deployment/topology-guide.md` → *Reading the replication backlog* for the healthy-versus-faulty signature. |
|
| `LocalDbOplogBacklog` | Consolidated site LocalDb | Unacked oplog depth (Prometheus `localdb_oplog_depth`). **Nullable — null means "no reading", NOT "connected with an empty backlog"** (a failed poll rendered as 0 would report a pair that cannot read its own oplog as perfectly healthy). **Expected transient spikes:** the daily `site_events` retention purge and the storage-cap trim are CDC-captured like any other write, so the backlog jumps by the batch size at purge time and drains as the peer acks it — see `Component-SiteEventLogging.md` → Storage and `docs/deployment/topology-guide.md` → *Reading the replication backlog* for the healthy-versus-faulty signature. |
|
||||||
|
|
||||||
## Reporting Protocol
|
## Reporting Protocol
|
||||||
|
|||||||
@@ -451,7 +451,7 @@ Site secrets ride the artifact deploy (design `docs/plans/2026-09-28-site-secret
|
|||||||
- **Isolation.** One row failing to apply/verify does not block any other row in the same pass.
|
- **Isolation.** One row failing to apply/verify does not block any other row in the same pass.
|
||||||
- **Cache invalidation.** A successful apply or delete invalidates the shared `ISecretResolver` cache before verifying, so the verify step (and any script/connection reading the secret) sees the fresh value immediately rather than waiting out the resolver's cache TTL.
|
- **Cache invalidation.** A successful apply or delete invalidates the shared `ISecretResolver` cache before verifying, so the verify step (and any script/connection reading the secret) sees the fresh value immediately rather than waiting out the resolver's cache TTL.
|
||||||
- **Never logs a secret value** — log lines and the reported `MirrorResult` carry only names, revisions, and `KekId`s.
|
- **Never logs a secret value** — log lines and the reported `MirrorResult` carry only names, revisions, and `KekId`s.
|
||||||
- **Mirror status on the health report (review finding M1).** A failure on the **standby** — above all a secret that will not decrypt because the standby's master key differs from central's — used to live only in that node's log, invisible until a failover broke its connections. After every pass the mirror records its node's current failure set in `secret_mirror_status`, but **only when the set (sorted name+message pairs) differs from what this process last wrote**, plus once on the first pass after start-up so a restart refreshes a stale row — never on every 15 s tick, because every write is a replicated CDC change. The write is best-effort: a failure logs a Warning (repeats at Debug), never affects the pass result, and is retried next pass. `SecretMirrorStatusReporter` (`SiteRuntime/Secrets/`, every site node, 30 s, immediate first probe) reads every node's row into `ISiteHealthCollector.SetSecretMirrorStatuses`, so the active node's report carries both nodes' status as `SiteHealthReport.SecretMirrorStatuses` (see Component-HealthMonitoring.md). On an **unreplicated** pair only the active node's own row is visible — the same replication prerequisite deployed secrets already have.
|
- **Mirror status on the health report (review finding M1).** A failure on the **standby** — above all a secret that will not decrypt because the standby's master key differs from central's — used to live only in that node's log, invisible until a failover broke its connections. After every pass the mirror records its node's current failure set in `secret_mirror_status`, but **only when the set (sorted name+message pairs) differs from what this process last wrote**, plus once on the first pass after start-up so a restart refreshes a stale row, plus at least every `SiteRuntime:SecretMirrorStatusRefreshInterval` (default **10 min**) as a heartbeat — never on every 15 s tick, because every write is a replicated CDC change. The write is best-effort: a failure logs a Warning (repeats at Debug), never affects the pass result, and is retried next pass. `SecretMirrorStatusReporter` (`SiteRuntime/Secrets/`, every site node, 30 s, immediate first probe) reads every node's row into `ISiteHealthCollector.SetSecretMirrorStatuses`, so the active node's report carries both nodes' status as `SiteHealthReport.SecretMirrorStatuses` (see Component-HealthMonitoring.md). On an **unreplicated** pair only the active node's own row is visible — the same replication prerequisite deployed secrets already have. **Stale rows are pruned:** the row is keyed by node name, so a decommissioned or renamed node used to leave its row (and any *Secret mirror failing* badge) behind forever. The reporter now leaves off the report, and deletes, any row whose `checked_at` is older than `SiteRuntime:SecretMirrorStatusStaleAfter` (default **1 h**, validated `≥ 2 ×` the refresh interval) — the delete is conditional on the `checked_at` it read, so a node that re-wrote its row in the meantime keeps it, and it replicates like any other write. A node down longer than that also drops off the list (its absence already shows in the cluster member count) and writes its row again on its first pass after it returns.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -13,7 +13,7 @@ namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
|
|||||||
/// and human-readable messages only — never a secret value or ciphertext.
|
/// and human-readable messages only — never a secret value or ciphertext.
|
||||||
/// </remarks>
|
/// </remarks>
|
||||||
/// <param name="NodeName">The reporting node's semantic name (<c>node-a</c>/<c>node-b</c>), or its host name when unconfigured.</param>
|
/// <param name="NodeName">The reporting node's semantic name (<c>node-a</c>/<c>node-b</c>), or its host name when unconfigured.</param>
|
||||||
/// <param name="CheckedAtUtc">When that node last recorded a CHANGE in its failure set (or its first pass after start-up) — not the time of its latest pass.</param>
|
/// <param name="CheckedAtUtc">When that node last wrote its row: on a CHANGE in its failure set, on its first pass after start-up, or on the periodic refresh (<c>SiteRuntime:SecretMirrorStatusRefreshInterval</c>, 10 min) — not the time of its latest pass. Rows older than <c>SecretMirrorStatusStaleAfter</c> are pruned.</param>
|
||||||
/// <param name="Failures">That node's current mirror failures; empty when the node's mirror is healthy.</param>
|
/// <param name="Failures">That node's current mirror failures; empty when the node's mirror is healthy.</param>
|
||||||
public sealed record SecretMirrorNodeStatus(
|
public sealed record SecretMirrorNodeStatus(
|
||||||
string NodeName,
|
string NodeName,
|
||||||
|
|||||||
@@ -1006,6 +1006,30 @@ public class SiteStorageService
|
|||||||
return results;
|
return results;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Deletes <paramref name="nodeName"/>'s <c>secret_mirror_status</c> row, but only while it
|
||||||
|
/// still carries <paramref name="checkedAtUtc"/> — so a row the node re-wrote after the caller
|
||||||
|
/// read it (it came back, or refreshed just in time) is never deleted. The delete replicates
|
||||||
|
/// like any other write.
|
||||||
|
/// </summary>
|
||||||
|
/// <param name="nodeName">The row key.</param>
|
||||||
|
/// <param name="checkedAtUtc">The <c>checked_at</c> the caller read and judged stale.</param>
|
||||||
|
/// <returns><see langword="true"/> when the row was deleted.</returns>
|
||||||
|
public async Task<bool> DeleteSecretMirrorStatusIfUnchangedAsync(string nodeName, DateTimeOffset checkedAtUtc)
|
||||||
|
{
|
||||||
|
ArgumentException.ThrowIfNullOrWhiteSpace(nodeName);
|
||||||
|
|
||||||
|
await using var connection = OpenConnection();
|
||||||
|
|
||||||
|
await using var command = connection.CreateCommand();
|
||||||
|
command.CommandText =
|
||||||
|
"DELETE FROM secret_mirror_status WHERE node_name = @nodeName AND checked_at = @checkedAt";
|
||||||
|
command.Parameters.AddWithValue("@nodeName", nodeName);
|
||||||
|
command.Parameters.AddWithValue("@checkedAt", checkedAtUtc.ToUniversalTime().ToString("O"));
|
||||||
|
|
||||||
|
return await command.ExecuteNonQueryAsync() > 0;
|
||||||
|
}
|
||||||
|
|
||||||
// ── Shared Script CRUD ──
|
// ── Shared Script CRUD ──
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
|
|||||||
@@ -163,7 +163,10 @@ public sealed record MirrorResult(
|
|||||||
/// name; <c>SecretMirrorStatusReporter</c> on the active node reads every node's row onto the
|
/// name; <c>SecretMirrorStatusReporter</c> on the active node reads every node's row onto the
|
||||||
/// health report. Because every write replicates via CDC, the row is written only when the
|
/// health report. Because every write replicates via CDC, the row is written only when the
|
||||||
/// failure set (compared as sorted name+message pairs) differs from what this process last
|
/// failure set (compared as sorted name+message pairs) differs from what this process last
|
||||||
/// wrote — plus once on the first pass after start-up, so a restart refreshes a stale row. The
|
/// wrote — plus once on the first pass after start-up, so a restart refreshes a stale row, and
|
||||||
|
/// at least every <see cref="SiteRuntimeOptions.SecretMirrorStatusRefreshInterval"/> so the
|
||||||
|
/// row's <c>checked_at</c> works as a heartbeat: <c>SecretMirrorStatusReporter</c> prunes the
|
||||||
|
/// row of a node that has stopped refreshing it (decommissioned or renamed). The
|
||||||
/// write is best-effort: a failure logs a Warning (repeats at Debug) and is retried on the next
|
/// write is best-effort: a failure logs a Warning (repeats at Debug) and is retried on the next
|
||||||
/// pass, and never affects the pass's own <see cref="MirrorResult"/>.
|
/// pass, and never affects the pass's own <see cref="MirrorResult"/>.
|
||||||
/// </para>
|
/// </para>
|
||||||
@@ -188,6 +191,9 @@ public sealed class DeployedSecretsMirror : BackgroundService
|
|||||||
/// </summary>
|
/// </summary>
|
||||||
private IReadOnlyList<SecretMirrorFailure>? _lastWrittenStatus;
|
private IReadOnlyList<SecretMirrorFailure>? _lastWrittenStatus;
|
||||||
|
|
||||||
|
/// <summary>When this process last wrote its <c>secret_mirror_status</c> row (heartbeat refresh).</summary>
|
||||||
|
private DateTimeOffset _lastWrittenStatusAt;
|
||||||
|
|
||||||
/// <summary>Whether a failed status write has already been logged at Warning (de-spam).</summary>
|
/// <summary>Whether a failed status write has already been logged at Warning (de-spam).</summary>
|
||||||
private bool _statusWriteFailureLogged;
|
private bool _statusWriteFailureLogged;
|
||||||
|
|
||||||
@@ -358,7 +364,8 @@ public sealed class DeployedSecretsMirror : BackgroundService
|
|||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// Writes this node's <c>secret_mirror_status</c> row when the pass's failure set differs
|
/// Writes this node's <c>secret_mirror_status</c> row when the pass's failure set differs
|
||||||
/// from the last one this process wrote (or nothing has been written yet since start-up).
|
/// from the last one this process wrote, nothing has been written yet since start-up, or the
|
||||||
|
/// last write is older than <see cref="SiteRuntimeOptions.SecretMirrorStatusRefreshInterval"/>.
|
||||||
/// Best-effort: never throws, so the pass result is unaffected by a status-write fault.
|
/// Best-effort: never throws, so the pass result is unaffected by a status-write fault.
|
||||||
/// See the class remarks (review finding M1).
|
/// See the class remarks (review finding M1).
|
||||||
/// </summary>
|
/// </summary>
|
||||||
@@ -370,15 +377,19 @@ public sealed class DeployedSecretsMirror : BackgroundService
|
|||||||
.ThenBy(f => f.Message, StringComparer.Ordinal)
|
.ThenBy(f => f.Message, StringComparer.Ordinal)
|
||||||
.ToList();
|
.ToList();
|
||||||
|
|
||||||
if (_lastWrittenStatus is not null && _lastWrittenStatus.SequenceEqual(current))
|
var now = _time.GetUtcNow();
|
||||||
|
if (_lastWrittenStatus is not null
|
||||||
|
&& _lastWrittenStatus.SequenceEqual(current)
|
||||||
|
&& now - _lastWrittenStatusAt < _options.Value.SecretMirrorStatusRefreshInterval)
|
||||||
{
|
{
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
await _storage.SetSecretMirrorStatusAsync(_nodeName, current, _time.GetUtcNow()).ConfigureAwait(false);
|
await _storage.SetSecretMirrorStatusAsync(_nodeName, current, now).ConfigureAwait(false);
|
||||||
_lastWrittenStatus = current;
|
_lastWrittenStatus = current;
|
||||||
|
_lastWrittenStatusAt = now;
|
||||||
_statusWriteFailureLogged = false;
|
_statusWriteFailureLogged = false;
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
using Microsoft.Extensions.Hosting;
|
using Microsoft.Extensions.Hosting;
|
||||||
using Microsoft.Extensions.Logging;
|
using Microsoft.Extensions.Logging;
|
||||||
|
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
|
||||||
using ZB.MOM.WW.ScadaBridge.HealthMonitoring;
|
using ZB.MOM.WW.ScadaBridge.HealthMonitoring;
|
||||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
|
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
|
||||||
|
|
||||||
@@ -25,6 +26,15 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Secrets;
|
|||||||
/// first probe, fixed cadence, exceptions logged and swallowed. A failed read leaves the
|
/// first probe, fixed cadence, exceptions logged and swallowed. A failed read leaves the
|
||||||
/// previous snapshot on the collector rather than clearing it.
|
/// previous snapshot on the collector rather than clearing it.
|
||||||
/// </para>
|
/// </para>
|
||||||
|
/// <para>
|
||||||
|
/// <b>Stale rows.</b> Every live node re-writes its row at least every
|
||||||
|
/// <see cref="SiteRuntimeOptions.SecretMirrorStatusRefreshInterval"/>, so a row whose
|
||||||
|
/// <c>checked_at</c> is older than <see cref="SiteRuntimeOptions.SecretMirrorStatusStaleAfter"/>
|
||||||
|
/// belongs to a node that is no longer running its mirror — decommissioned, renamed (the row is
|
||||||
|
/// keyed by node name), or down for that long. Such a row is left off the report and deleted
|
||||||
|
/// (only while it still carries the stale <c>checked_at</c>, so a node that came back in the
|
||||||
|
/// meantime keeps its fresh row); a returning node writes its row again on its first pass.
|
||||||
|
/// </para>
|
||||||
/// </remarks>
|
/// </remarks>
|
||||||
public sealed class SecretMirrorStatusReporter : BackgroundService
|
public sealed class SecretMirrorStatusReporter : BackgroundService
|
||||||
{
|
{
|
||||||
@@ -35,22 +45,33 @@ public sealed class SecretMirrorStatusReporter : BackgroundService
|
|||||||
private readonly SiteStorageService _storage;
|
private readonly SiteStorageService _storage;
|
||||||
private readonly ILogger<SecretMirrorStatusReporter> _logger;
|
private readonly ILogger<SecretMirrorStatusReporter> _logger;
|
||||||
private readonly TimeSpan _pollInterval;
|
private readonly TimeSpan _pollInterval;
|
||||||
|
private readonly TimeSpan _staleAfter;
|
||||||
|
private readonly TimeProvider _time;
|
||||||
|
|
||||||
/// <summary>Initializes a new instance of <see cref="SecretMirrorStatusReporter"/>.</summary>
|
/// <summary>Initializes a new instance of <see cref="SecretMirrorStatusReporter"/>.</summary>
|
||||||
/// <param name="collector">The site health collector receiving the statuses.</param>
|
/// <param name="collector">The site health collector receiving the statuses.</param>
|
||||||
/// <param name="storage">Site storage exposing <c>secret_mirror_status</c>.</param>
|
/// <param name="storage">Site storage exposing <c>secret_mirror_status</c>.</param>
|
||||||
/// <param name="logger">Logger instance.</param>
|
/// <param name="logger">Logger instance.</param>
|
||||||
/// <param name="pollInterval">Poll interval override; defaults to <see cref="DefaultPollInterval"/> (30 s).</param>
|
/// <param name="pollInterval">Poll interval override; defaults to <see cref="DefaultPollInterval"/> (30 s).</param>
|
||||||
|
/// <param name="staleAfter">
|
||||||
|
/// Age after which a row is pruned; defaults to the
|
||||||
|
/// <see cref="SiteRuntimeOptions.SecretMirrorStatusStaleAfter"/> default (1 hour).
|
||||||
|
/// </param>
|
||||||
|
/// <param name="time">Clock used to age rows; defaults to <see cref="TimeProvider.System"/>.</param>
|
||||||
public SecretMirrorStatusReporter(
|
public SecretMirrorStatusReporter(
|
||||||
ISiteHealthCollector collector,
|
ISiteHealthCollector collector,
|
||||||
SiteStorageService storage,
|
SiteStorageService storage,
|
||||||
ILogger<SecretMirrorStatusReporter> logger,
|
ILogger<SecretMirrorStatusReporter> logger,
|
||||||
TimeSpan? pollInterval = null)
|
TimeSpan? pollInterval = null,
|
||||||
|
TimeSpan? staleAfter = null,
|
||||||
|
TimeProvider? time = null)
|
||||||
{
|
{
|
||||||
_collector = collector ?? throw new ArgumentNullException(nameof(collector));
|
_collector = collector ?? throw new ArgumentNullException(nameof(collector));
|
||||||
_storage = storage ?? throw new ArgumentNullException(nameof(storage));
|
_storage = storage ?? throw new ArgumentNullException(nameof(storage));
|
||||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||||
_pollInterval = pollInterval ?? DefaultPollInterval;
|
_pollInterval = pollInterval ?? DefaultPollInterval;
|
||||||
|
_staleAfter = staleAfter ?? new SiteRuntimeOptions().SecretMirrorStatusStaleAfter;
|
||||||
|
_time = time ?? TimeProvider.System;
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <inheritdoc />
|
/// <inheritdoc />
|
||||||
@@ -74,7 +95,8 @@ public sealed class SecretMirrorStatusReporter : BackgroundService
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// Reads every node's mirror status and pushes it onto the collector; never throws.
|
/// Reads every node's mirror status, prunes rows older than the stale threshold, and pushes
|
||||||
|
/// the rest onto the collector; never throws.
|
||||||
/// Public so a test can drive one deterministic probe.
|
/// Public so a test can drive one deterministic probe.
|
||||||
/// </summary>
|
/// </summary>
|
||||||
/// <returns>A task that completes when the probe has finished.</returns>
|
/// <returns>A task that completes when the probe has finished.</returns>
|
||||||
@@ -83,7 +105,20 @@ public sealed class SecretMirrorStatusReporter : BackgroundService
|
|||||||
try
|
try
|
||||||
{
|
{
|
||||||
var statuses = await _storage.GetSecretMirrorStatusesAsync().ConfigureAwait(false);
|
var statuses = await _storage.GetSecretMirrorStatusesAsync().ConfigureAwait(false);
|
||||||
_collector.SetSecretMirrorStatuses(statuses);
|
var cutoff = _time.GetUtcNow() - _staleAfter;
|
||||||
|
var live = new List<SecretMirrorNodeStatus>(statuses.Count);
|
||||||
|
foreach (var status in statuses)
|
||||||
|
{
|
||||||
|
if (status.CheckedAtUtc >= cutoff)
|
||||||
|
{
|
||||||
|
live.Add(status);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
await PruneAsync(status).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
|
||||||
|
_collector.SetSecretMirrorStatuses(live);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
@@ -92,4 +127,30 @@ public sealed class SecretMirrorStatusReporter : BackgroundService
|
|||||||
_logger.LogWarning(ex, "SecretMirrorStatusReporter probe failed; next tick will retry.");
|
_logger.LogWarning(ex, "SecretMirrorStatusReporter probe failed; next tick will retry.");
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Deletes one stale row; a failed delete is logged and retried on the next probe (the row
|
||||||
|
/// stays off the report either way).
|
||||||
|
/// </summary>
|
||||||
|
private async Task PruneAsync(SecretMirrorNodeStatus status)
|
||||||
|
{
|
||||||
|
try
|
||||||
|
{
|
||||||
|
if (await _storage.DeleteSecretMirrorStatusIfUnchangedAsync(status.NodeName, status.CheckedAtUtc)
|
||||||
|
.ConfigureAwait(false))
|
||||||
|
{
|
||||||
|
_logger.LogInformation(
|
||||||
|
"Removed the secret mirror status row for node {NodeName}: not refreshed since {CheckedAtUtc:O} " +
|
||||||
|
"(older than {StaleAfter}); the node is decommissioned, renamed or down. It is re-created " +
|
||||||
|
"on the node's next mirror pass if it returns.",
|
||||||
|
status.NodeName, status.CheckedAtUtc, _staleAfter);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
_logger.LogWarning(ex,
|
||||||
|
"Failed to remove the stale secret mirror status row for node {NodeName}; next tick will retry.",
|
||||||
|
status.NodeName);
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -145,7 +145,9 @@ public static class ServiceCollectionExtensions
|
|||||||
services.AddHostedService(sp => new SecretMirrorStatusReporter(
|
services.AddHostedService(sp => new SecretMirrorStatusReporter(
|
||||||
sp.GetRequiredService<HealthMonitoring.ISiteHealthCollector>(),
|
sp.GetRequiredService<HealthMonitoring.ISiteHealthCollector>(),
|
||||||
sp.GetRequiredService<SiteStorageService>(),
|
sp.GetRequiredService<SiteStorageService>(),
|
||||||
sp.GetRequiredService<ILogger<SecretMirrorStatusReporter>>()));
|
sp.GetRequiredService<ILogger<SecretMirrorStatusReporter>>(),
|
||||||
|
staleAfter: sp.GetRequiredService<IOptions<SiteRuntimeOptions>>().Value.SecretMirrorStatusStaleAfter,
|
||||||
|
time: sp.GetRequiredService<TimeProvider>()));
|
||||||
|
|
||||||
return services;
|
return services;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -274,4 +274,26 @@ public class SiteRuntimeOptions
|
|||||||
/// cadence. Default: 15 seconds.
|
/// cadence. Default: 15 seconds.
|
||||||
/// </summary>
|
/// </summary>
|
||||||
public TimeSpan DeployedSecretsMirrorInterval { get; set; } = TimeSpan.FromSeconds(15);
|
public TimeSpan DeployedSecretsMirrorInterval { get; set; } = TimeSpan.FromSeconds(15);
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// How often <see cref="Secrets.DeployedSecretsMirror"/> re-writes this node's
|
||||||
|
/// <c>secret_mirror_status</c> row even when its failure set has not changed — the row's
|
||||||
|
/// <c>checked_at</c> doubles as a liveness heartbeat, so a row that stops being refreshed
|
||||||
|
/// can be recognised as belonging to a node that is gone (decommissioned, renamed, or down)
|
||||||
|
/// and pruned by <see cref="Secrets.SecretMirrorStatusReporter"/> after
|
||||||
|
/// <see cref="SecretMirrorStatusStaleAfter"/>. Each refresh is one replicated row write.
|
||||||
|
/// Default: 10 minutes.
|
||||||
|
/// </summary>
|
||||||
|
public TimeSpan SecretMirrorStatusRefreshInterval { get; set; } = TimeSpan.FromMinutes(10);
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Age (by <c>checked_at</c>) after which a <c>secret_mirror_status</c> row is treated as
|
||||||
|
/// belonging to a node that no longer exists: <see cref="Secrets.SecretMirrorStatusReporter"/>
|
||||||
|
/// leaves it off the health report and deletes it (the delete replicates like any other
|
||||||
|
/// write). A live node re-writes its row every <see cref="SecretMirrorStatusRefreshInterval"/>
|
||||||
|
/// and once at start-up, so only a node that has stopped running its mirror for this long
|
||||||
|
/// loses its row — and gets it back on its first pass after it returns. Must be at least
|
||||||
|
/// twice <see cref="SecretMirrorStatusRefreshInterval"/>. Default: 1 hour.
|
||||||
|
/// </summary>
|
||||||
|
public TimeSpan SecretMirrorStatusStaleAfter { get; set; } = TimeSpan.FromHours(1);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -136,5 +136,13 @@ public sealed class SiteRuntimeOptionsValidator : OptionsValidatorBase<SiteRunti
|
|||||||
$"ScadaBridge:SiteRuntime:DeployedSecretsMirrorInterval must be a positive duration " +
|
$"ScadaBridge:SiteRuntime:DeployedSecretsMirrorInterval must be a positive duration " +
|
||||||
$"(was {options.DeployedSecretsMirrorInterval}); it is the DeployedSecretsMirror " +
|
$"(was {options.DeployedSecretsMirrorInterval}); it is the DeployedSecretsMirror " +
|
||||||
"reconcile PeriodicTimer period.");
|
"reconcile PeriodicTimer period.");
|
||||||
|
builder.RequireThat(options.SecretMirrorStatusRefreshInterval > TimeSpan.Zero,
|
||||||
|
$"ScadaBridge:SiteRuntime:SecretMirrorStatusRefreshInterval must be a positive duration " +
|
||||||
|
$"(was {options.SecretMirrorStatusRefreshInterval}); it is how often each node re-writes its " +
|
||||||
|
"secret_mirror_status row.");
|
||||||
|
builder.RequireThat(options.SecretMirrorStatusStaleAfter >= options.SecretMirrorStatusRefreshInterval * 2,
|
||||||
|
$"ScadaBridge:SiteRuntime:SecretMirrorStatusStaleAfter (was {options.SecretMirrorStatusStaleAfter}) must be " +
|
||||||
|
$"at least twice SecretMirrorStatusRefreshInterval (was {options.SecretMirrorStatusRefreshInterval}); " +
|
||||||
|
"otherwise a live node's row could be pruned between two refreshes.");
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -695,6 +695,26 @@ public class DeployedSecretsMirrorTests : IDisposable
|
|||||||
Assert.Contains("kek-central", failure.Message);
|
Assert.Contains("kek-central", failure.Message);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Status_UnchangedFailureSet_IsRefreshedOnceTheRefreshIntervalHasPassed()
|
||||||
|
{
|
||||||
|
// The row's checked_at is a heartbeat: SecretMirrorStatusReporter prunes rows that stop
|
||||||
|
// being refreshed, so a live node must re-write an unchanged row every refresh interval.
|
||||||
|
var time = new FakeTimeProvider(new DateTimeOffset(2026, 9, 28, 6, 0, 0, TimeSpan.Zero));
|
||||||
|
var (mirror, storage, _, _, _) = await CreateAsync(time: time, nodeIdentity: new FixedNodeIdentity("node-b"));
|
||||||
|
|
||||||
|
await mirror.ReconcileAsync(CancellationToken.None);
|
||||||
|
var first = Assert.Single(await storage.GetSecretMirrorStatusesAsync()).CheckedAtUtc;
|
||||||
|
|
||||||
|
time.Now = first + new SiteRuntimeOptions().SecretMirrorStatusRefreshInterval - TimeSpan.FromSeconds(1);
|
||||||
|
await mirror.ReconcileAsync(CancellationToken.None);
|
||||||
|
Assert.Equal(first, Assert.Single(await storage.GetSecretMirrorStatusesAsync()).CheckedAtUtc);
|
||||||
|
|
||||||
|
time.Now = first + new SiteRuntimeOptions().SecretMirrorStatusRefreshInterval;
|
||||||
|
await mirror.ReconcileAsync(CancellationToken.None);
|
||||||
|
Assert.Equal(time.Now, Assert.Single(await storage.GetSecretMirrorStatusesAsync()).CheckedAtUtc);
|
||||||
|
}
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
public async Task Status_FailureSetChange_RewritesTheRow_AndClearsWhenHealed()
|
public async Task Status_FailureSetChange_RewritesTheRow_AndClearsWhenHealed()
|
||||||
{
|
{
|
||||||
|
|||||||
+58
-1
@@ -40,7 +40,8 @@ public class SecretMirrorStatusReporterTests : IAsyncLifetime, IDisposable
|
|||||||
"node-b", [new SecretMirrorFailure("OPC-PWD", "did not decrypt on this node")], at);
|
"node-b", [new SecretMirrorFailure("OPC-PWD", "did not decrypt on this node")], at);
|
||||||
var collector = new SiteHealthCollector();
|
var collector = new SiteHealthCollector();
|
||||||
var reporter = new SecretMirrorStatusReporter(
|
var reporter = new SecretMirrorStatusReporter(
|
||||||
collector, _storage, NullLogger<SecretMirrorStatusReporter>.Instance);
|
collector, _storage, NullLogger<SecretMirrorStatusReporter>.Instance,
|
||||||
|
time: new FixedTime(at.AddMinutes(1)));
|
||||||
|
|
||||||
await reporter.ProbeAsync();
|
await reporter.ProbeAsync();
|
||||||
var report = collector.CollectReport("site-a");
|
var report = collector.CollectReport("site-a");
|
||||||
@@ -92,4 +93,60 @@ public class SecretMirrorStatusReporterTests : IAsyncLifetime, IDisposable
|
|||||||
Assert.Single(collector.CollectReport("site-a").SecretMirrorStatuses!);
|
Assert.Single(collector.CollectReport("site-a").SecretMirrorStatuses!);
|
||||||
Assert.Single(collector.CollectReport("site-a").SecretMirrorStatuses!);
|
Assert.Single(collector.CollectReport("site-a").SecretMirrorStatuses!);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ── Stale rows (decommissioned / renamed nodes) ──
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Probe_RowOlderThanStaleAfter_IsLeftOffTheReport_AndDeleted()
|
||||||
|
{
|
||||||
|
var now = new DateTimeOffset(2026, 9, 29, 12, 0, 0, TimeSpan.Zero);
|
||||||
|
await _storage.SetSecretMirrorStatusAsync("node-a", [], now.AddMinutes(-5));
|
||||||
|
await _storage.SetSecretMirrorStatusAsync(
|
||||||
|
"node-old", [new SecretMirrorFailure("OPC-PWD", "did not decrypt on this node")], now.AddHours(-2));
|
||||||
|
var collector = new SiteHealthCollector();
|
||||||
|
var reporter = new SecretMirrorStatusReporter(
|
||||||
|
collector, _storage, NullLogger<SecretMirrorStatusReporter>.Instance,
|
||||||
|
staleAfter: TimeSpan.FromHours(1), time: new FixedTime(now));
|
||||||
|
|
||||||
|
await reporter.ProbeAsync();
|
||||||
|
|
||||||
|
Assert.Equal(["node-a"], collector.CollectReport("site-a").SecretMirrorStatuses!.Select(s => s.NodeName));
|
||||||
|
Assert.Equal(["node-a"], (await _storage.GetSecretMirrorStatusesAsync()).Select(s => s.NodeName));
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Probe_RowExactlyAtTheCutoff_IsKept()
|
||||||
|
{
|
||||||
|
var now = new DateTimeOffset(2026, 9, 29, 12, 0, 0, TimeSpan.Zero);
|
||||||
|
await _storage.SetSecretMirrorStatusAsync("node-b", [], now.AddHours(-1));
|
||||||
|
var collector = new SiteHealthCollector();
|
||||||
|
var reporter = new SecretMirrorStatusReporter(
|
||||||
|
collector, _storage, NullLogger<SecretMirrorStatusReporter>.Instance,
|
||||||
|
staleAfter: TimeSpan.FromHours(1), time: new FixedTime(now));
|
||||||
|
|
||||||
|
await reporter.ProbeAsync();
|
||||||
|
|
||||||
|
Assert.Single(collector.CollectReport("site-a").SecretMirrorStatuses!);
|
||||||
|
Assert.Single(await _storage.GetSecretMirrorStatusesAsync());
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task ConditionalDelete_DoesNotRemoveARowTheNodeRewroteAfterTheRead()
|
||||||
|
{
|
||||||
|
// The reporter read a stale checked_at, but the node came back and re-wrote its row
|
||||||
|
// before the delete ran: the fresh row must survive.
|
||||||
|
var stale = new DateTimeOffset(2026, 9, 29, 9, 0, 0, TimeSpan.Zero);
|
||||||
|
await _storage.SetSecretMirrorStatusAsync("node-b", [], stale);
|
||||||
|
await _storage.SetSecretMirrorStatusAsync("node-b", [], stale.AddHours(3));
|
||||||
|
|
||||||
|
Assert.False(await _storage.DeleteSecretMirrorStatusIfUnchangedAsync("node-b", stale));
|
||||||
|
Assert.Single(await _storage.GetSecretMirrorStatusesAsync());
|
||||||
|
Assert.True(await _storage.DeleteSecretMirrorStatusIfUnchangedAsync("node-b", stale.AddHours(3)));
|
||||||
|
Assert.Empty(await _storage.GetSecretMirrorStatusesAsync());
|
||||||
|
}
|
||||||
|
|
||||||
|
private sealed class FixedTime(DateTimeOffset now) : TimeProvider
|
||||||
|
{
|
||||||
|
public override DateTimeOffset GetUtcNow() => now;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -243,4 +243,26 @@ public class SiteRuntimeOptionsValidatorTests
|
|||||||
Assert.True(result.Failed);
|
Assert.True(result.Failed);
|
||||||
Assert.Contains("ScriptFileAccess:MaxFileBytes", result.FailureMessage);
|
Assert.Contains("ScriptFileAccess:MaxFileBytes", result.FailureMessage);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void ZeroSecretMirrorStatusRefreshInterval_IsRejected()
|
||||||
|
{
|
||||||
|
var result = Validate(new SiteRuntimeOptions { SecretMirrorStatusRefreshInterval = TimeSpan.Zero });
|
||||||
|
|
||||||
|
Assert.True(result.Failed);
|
||||||
|
Assert.Contains("SecretMirrorStatusRefreshInterval", result.FailureMessage);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void SecretMirrorStatusStaleAfter_UnderTwoRefreshes_IsRejected()
|
||||||
|
{
|
||||||
|
var result = Validate(new SiteRuntimeOptions
|
||||||
|
{
|
||||||
|
SecretMirrorStatusRefreshInterval = TimeSpan.FromMinutes(10),
|
||||||
|
SecretMirrorStatusStaleAfter = TimeSpan.FromMinutes(15),
|
||||||
|
});
|
||||||
|
|
||||||
|
Assert.True(result.Failed);
|
||||||
|
Assert.Contains("SecretMirrorStatusStaleAfter", result.FailureMessage);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user