Compare commits
27 Commits
2aede140e2
...
23a139a46c
| Author | SHA1 | Date | |
|---|---|---|---|
| 23a139a46c | |||
| d641186397 | |||
| 67a3950f74 | |||
| b1b49bb36c | |||
| 6f1787abe1 | |||
| 9835ef8be3 | |||
| ad19d4b158 | |||
| 1444f85919 | |||
| 0ccf68951d | |||
| bc29dc412b | |||
| 2bac13170c | |||
| 52cd4fb096 | |||
| 838fb4177b | |||
| 4d7d6456e3 | |||
| 25c25ccef9 | |||
| a35ee663c8 | |||
| 437b02a43c | |||
| 00c89aff12 | |||
| 75b2a7244c | |||
| a57c36f504 | |||
| 35235ecfed | |||
| ebc55c5f72 | |||
| 8aecd55db5 | |||
| 738e458755 | |||
| 01cb1f5652 | |||
| bf7d2db0b1 | |||
| b6287ae307 |
@@ -90,7 +90,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
|
||||
- DCL write failures returned synchronously to calling script.
|
||||
- Tag path resolution retried periodically for devices still booting.
|
||||
- Static attribute writes persisted to local SQLite (survive restart/failover, reset on redeployment).
|
||||
- **Consolidated site database (LocalDb Phase 1 + 2, complete 2026-07-20).** Ten tables now live in ONE `ZB.MOM.WW.LocalDb`-managed SQLite file (the `SiteLocalDbSetup.ReplicatedTables` list), configured by the **required** `LocalDb:Path` (`/app/data/site-localdb.db` on the rig; validated with `ValidateOnStart`, so a site config missing it fails to boot). Consequences worth knowing:
|
||||
- **Consolidated site database (LocalDb Phase 1 + 2, complete 2026-07-20).** Eleven tables now live in ONE `ZB.MOM.WW.LocalDb`-managed SQLite file (the `SiteLocalDbSetup.ReplicatedTables` list — the 11th, `deployed_secrets`, was added 2026-09-28 by the site-secrets-via-artifact-deploy change, see *Templates & Deployment* below), configured by the **required** `LocalDb:Path` (`/app/data/site-localdb.db` on the rig; validated with `ValidateOnStart`, so a site config missing it fails to boot). Consequences worth knowing:
|
||||
- `site_events.id` changed from autoincrement INTEGER to an application-minted **GUID**. Last-writer-wins keys on the primary key, so two nodes independently minting `id=1,2,3…` would destroy each other's events rather than merge them. The event-log read path uses a composite `(timestamp, id)` keyset cursor with an **opaque string** continuation token; `EventLogEntry.Id` and both `ContinuationToken`s are `string`/`string?` on the site↔central Akka DTOs.
|
||||
- `ScadaBridge:OperationTracking:ConnectionString`, `ScadaBridge:SiteEventLog:DatabasePath` and — as of Phase 2 — `ScadaBridge:StoreAndForward:SqliteDbPath` + `ScadaBridge:Database:SiteDbPath` are all **migration-only** — nothing reads them but `SiteLocalDbLegacyMigrator`, which copies a pre-Phase-1 file in once (deterministic `mig-{NodeName}-{legacyId}` event ids, `INSERT OR IGNORE`, runs AFTER `RegisterReplicated` so migrated rows replicate) and renames it `.migrated`. Delete the keys once a node has migrated.
|
||||
- This incidentally fixes a data-loss bug: both legacy databases defaulted to CWD-relative paths **outside** the mounted volume and were discarded on every container recreate.
|
||||
@@ -101,10 +101,10 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
|
||||
- **`notification_lists` and `smtp_configurations` are created but deliberately NOT registered.** They are permanently empty on a site (no writer since 2026-07-10, the migrator skips them, the active-node purge keeps them empty), and registering them would open a standing replication channel whose only historical payload was plaintext SMTP passwords. Pinned by a security-named test, and verified live: those two tables have **no CDC triggers** on either rig node.
|
||||
- **Operational constraints (read before upgrading a site pair):** stop and start both nodes TOGETHER — rolling one at a time is no longer supported, since the legacy `SfBufferSnapshot` compatibility handler went with the replicator. And a node offline longer than `LocalDb:Replication:TombstoneRetention` (default 7 days) can resurrect deleted rows on rejoin. See `docs/deployment/topology-guide.md`.
|
||||
- **Batching is by BYTE BUDGET as of LocalDb 0.2.0** — `LocalDb:Replication:MaxBatchBytes` (default **2 MB**, sized under the 4 MB gRPC cap) bounds a delta/snapshot message by summed serialized size via a per-message split in `SyncSession.PumpLoopAsync`, with `MaxBatchSize` demoted to a secondary row cap; a single row over budget is sent alone rather than stalling the stream. **The rig's old `MaxBatchSize = 16` pin (a hand-computed byte-budget proxy: ~70 KB worst-case `config_json` x the 500 default is ~35 MB) is retired, but `MaxBatchSize` is NOT fully redundant with `MaxBatchBytes`** — it also bounds the separate DB READ page in `OplogStore.ReadBatchAboveAsync`/`SnapshotStreamer`, which materializes the whole page into memory *before* the byte-budget split runs, so an unset (500-default) `MaxBatchSize` still lets a reconnect drain transiently allocate ~35 MB per read even though every wire message stays under `MaxBatchBytes` (arch-review adversarial finding F2). Both site-a nodes on `docker/` therefore pin an explicit `"MaxBatchSize": 64` to bound that transient allocation, while `MaxBatchBytes` stays unset (its 2 MB default) to bound the wire message; site-b/site-c stay unreplicated so the key doesn't apply there.
|
||||
- **CDC registration is conditional, and both directions self-heal at boot** (arch-review WP1.3 + WP3.3, `Host/SiteLocalDbSetup.cs`). Capture triggers are installed only when this node has replication configured — `PeerAddress` **or** `ApiKey`, an OR because only the dialling half sets `PeerAddress` while the passive half carries the key alone. An unreplicated node calls **`DeregisterReplicated`** on all ten tables at boot, dropping triggers an earlier build left behind and pruning their oplog/row-version rows (idempotent; logs once at Information when something was actually cleaned). A replicating node registers with **`baselineExistingRows: true`**, which seeds `__localdb_row_version` for pre-existing rows at the LWW floor (HLC `0`, this node's id) and flags a snapshot resync — so turning replication ON for a site that has been running without it now converges on the rows already in the file instead of only on writes made after the restart. Deregistration must be symmetric (the handshake compares registered-table digests fail-closed), so replication is a both-nodes-together change in either direction. LocalDb 0.2.0 also makes backlog depth O(1) and drops the unused `__localdb_oplog_hlc` index; the on-disk bookkeeping schema goes to v2, upgraded in place on open, with no wire change (0.1.x peers still sync).
|
||||
- **CDC registration is conditional, and both directions self-heal at boot** (arch-review WP1.3 + WP3.3, `Host/SiteLocalDbSetup.cs`). Capture triggers are installed only when this node has replication configured — `PeerAddress` **or** `ApiKey`, an OR because only the dialling half sets `PeerAddress` while the passive half carries the key alone. An unreplicated node calls **`DeregisterReplicated`** on all eleven tables at boot, dropping triggers an earlier build left behind and pruning their oplog/row-version rows (idempotent; logs once at Information when something was actually cleaned). A replicating node registers with **`baselineExistingRows: true`**, which seeds `__localdb_row_version` for pre-existing rows at the LWW floor (HLC `0`, this node's id) and flags a snapshot resync — so turning replication ON for a site that has been running without it now converges on the rows already in the file instead of only on writes made after the restart. Deregistration must be symmetric (the handshake compares registered-table digests fail-closed), so replication is a both-nodes-together change in either direction. LocalDb 0.2.0 also makes backlog depth O(1) and drops the unused `__localdb_oplog_hlc` index; the on-disk bookkeeping schema goes to v2, upgraded in place on open, with no wire change (0.1.x peers still sync).
|
||||
- **LocalDb 0.3.0 (2026-08-15) makes that snapshot flag PER-TABLE** — the residual 0.2.1 explicitly deferred. On-disk bookkeeping schema goes to **v3** (a new `__localdb_snapshot_state` table, upgraded in place on open); still wire-compatible, and `lib_schema_version` deliberately stays `1` because the handshake compares it fail-closed for *equality*. The extension is negotiated by **capability** (`Handshake.supports_partial_snapshot`, `SnapshotBegin.tables` where **empty == all tables**), so a peer that does not advertise it is served a full snapshot — a scoped one would make it advance `last_applied_remote_seq` past deltas it never received. For the same reason **a partial snapshot moves no watermark on either side**; the overlapping deltas are still pumped and absorbed by LWW. Snapshot debt now clears on the receiver's new `SnapshotAck` rather than on enqueue, so a crash mid-snapshot re-streams the same scope. **`SiteLocalDbSetup.cs` needed no restructuring** — it already calls `RegisterReplicated` once per table in a loop, which is exactly what per-table flagging keys off. The consequence to know: **adding an eleventh table to `ReplicatedTables` on an already-replicating site now snapshots that one table**, where through 0.2.x the same edit re-streamed all eleven in full in both directions. First-boot behaviour is unchanged (all ten seed at once ⇒ the flagged set is every registered table ⇒ an ordinary full snapshot), and so is upgrading the rig in place (the ten tables are already ledgered, nothing seeds, nothing is flagged). A future "register the whole list in one call" API would silently give the coarse behaviour back.
|
||||
- **LocalDb 0.3.2 (2026-08-19, ScadaBridge arch-review P4-1 + P4-3; wire-compatible, no schema change):** (1) the sync pump now **faults the live session when a cap-prune invalidates its position** (horizon check on the oplog's min seq before each read batch + on the batch head after it; throws the new public `LocalDbOplogGapException` through the 0.3.1 fault-teardown) — previously a slow-but-alive peer silently skipped the pruned gap and the pair diverged until the next reconnect; (2) **transport keepalives both sides** — client `SocketsHttpHandler` pings via new validated `LocalDb:Replication:KeepAlivePingDelay`/`KeepAlivePingTimeout` (defaults 60s/20s, `Always` policy), and ScadaBridge's site gRPC listener sets Kestrel h2 keepalive (`Host/SiteKestrelSetup.cs` — server-wide, also governs the metrics port); (3) the server's single-stream guard is now **newest-wins with an awaited handover** — a fresh authenticated `Sync` cancels a dead/wedged incumbent and *waits* for it to finish (`TakeoverTimeout` 10s) before starting, so a half-open link can no longer wedge the pair behind `AlreadyExists` while the zombie side reports `LocalDbReplicationConnected=true`.
|
||||
- **LocalDb 0.3.3 (2026-08-20, ScadaBridge arch-review P4-7; wire-compatible, no schema change — superseded by 0.3.4 below in the same bullet):** two latent `LwwApplier` hazards closed before any table could hit them. (1) **`RegisterReplicated` now REFUSES a table that is the parent of a foreign key with a row-mutating `ON DELETE` action (`CASCADE`/`SET NULL`/`SET DEFAULT`) from any other table** (`LocalDbRegistrationException` naming `child.column ON DELETE <action>`; `RESTRICT`/`NO ACTION` still allowed) — the applier's `INSERT OR REPLACE` runs with `foreign_keys=ON`, so the implicit delete fired those actions while CDC capture was suppressed, i.e. child rows changed on one node and never replicated. Registration-time only: SQLite can't add an FK to an existing table, so a later-created child is caught at the next boot's registration. None of ScadaBridge's ten tables has an FK (verified on the live rig file: zero `REFERENCES` across 29 `CREATE TABLE`s) — **keep it that way; a cascading-FK parent can never be one of `ReplicatedTables`** (register row 105). (2) The per-entry apply (delete-or-upsert + row-version ledger write) is now **atomic under a `SAVEPOINT`** — a fault between the two statements used to leave a deleted row with a live ledger entry, which `SnapshotStreamer` warns-and-skips forever; poison-catch semantics unchanged (narrow `SqliteException`, per-entry dead-letter, batch continues). Cost: two extra in-memory statements per applied entry on a backlog drain. **Both are pinned here** (2026-08-20): `SiteLocalDbReplicatedTableFkGuardTests` asserts at build time that no `ReplicatedTables` entry has an inbound row-mutating FK (register row 105), and **pre-2026-08-20 site files must be VACUUMed once offline** to get `auto_vacuum=INCREMENTAL` — `docker/vacuum-site-localdb.sh` (pair stopped together; done on all three rig pairs; topology guide → *one-off offline VACUUM*; row 104). **LocalDb 0.3.4 (2026-08-20, register row 106; wire-compatible, sender-side only):** a converged pair whose sender oplog was ack-pruned to empty used to stamp snapshots `as_of_seq 0`, so a peer with a zero watermark (a rebuilt node — the rig's site-a-a since 2026-08-15) received a FULL snapshot on every reconnect until the sender wrote again (127 k rows per restart, observed). The as-of now floors at `last_acked_seq`; proven live — second restart after the pin exchanged zero snapshots. Lesson for rebuilding a node from its peer: the first reconnect snapshots, the second must not.
|
||||
- **LocalDb 0.3.3 (2026-08-20, ScadaBridge arch-review P4-7; wire-compatible, no schema change — superseded by 0.3.4 below in the same bullet):** two latent `LwwApplier` hazards closed before any table could hit them. (1) **`RegisterReplicated` now REFUSES a table that is the parent of a foreign key with a row-mutating `ON DELETE` action (`CASCADE`/`SET NULL`/`SET DEFAULT`) from any other table** (`LocalDbRegistrationException` naming `child.column ON DELETE <action>`; `RESTRICT`/`NO ACTION` still allowed) — the applier's `INSERT OR REPLACE` runs with `foreign_keys=ON`, so the implicit delete fired those actions while CDC capture was suppressed, i.e. child rows changed on one node and never replicated. Registration-time only: SQLite can't add an FK to an existing table, so a later-created child is caught at the next boot's registration. None of ScadaBridge's eleven tables has an FK (verified on the live rig file: zero `REFERENCES` across 29 `CREATE TABLE`s) — **keep it that way; a cascading-FK parent can never be one of `ReplicatedTables`** (register row 105). (2) The per-entry apply (delete-or-upsert + row-version ledger write) is now **atomic under a `SAVEPOINT`** — a fault between the two statements used to leave a deleted row with a live ledger entry, which `SnapshotStreamer` warns-and-skips forever; poison-catch semantics unchanged (narrow `SqliteException`, per-entry dead-letter, batch continues). Cost: two extra in-memory statements per applied entry on a backlog drain. **Both are pinned here** (2026-08-20): `SiteLocalDbReplicatedTableFkGuardTests` asserts at build time that no `ReplicatedTables` entry has an inbound row-mutating FK (register row 105), and **pre-2026-08-20 site files must be VACUUMed once offline** to get `auto_vacuum=INCREMENTAL` — `docker/vacuum-site-localdb.sh` (pair stopped together; done on all three rig pairs; topology guide → *one-off offline VACUUM*; row 104). **LocalDb 0.3.4 (2026-08-20, register row 106; wire-compatible, sender-side only):** a converged pair whose sender oplog was ack-pruned to empty used to stamp snapshots `as_of_seq 0`, so a peer with a zero watermark (a rebuilt node — the rig's site-a-a since 2026-08-15) received a FULL snapshot on every reconnect until the sender wrote again (127 k rows per restart, observed). The as-of now floors at `last_acked_seq`; proven live — second restart after the pin exchanged zero snapshots. Lesson for rebuilding a node from its peer: the first reconnect snapshots, the second must not.
|
||||
- **LocalDb 0.2.1** (two adversarial-review findings against 0.2.0, wire-compatible, no schema change) fixes: (1) `DeregisterReplicated` was deleting the HLC clock's only durable crash anchor along with the table rows — an ungraceful exit (crash, SIGKILL, power loss) on a fully-deregistered file left nothing to recover from, so the reopened clock could restart from wall-clock time and re-issue stamps below HLCs peers already hold (silently discarded by LWW); the current clock is now flushed inside the deregistration transaction, before the prune. (2) The sync inbox was an **unbounded** channel, so inbound memory was a function of what the peer sends — a mutual full snapshot (both sides of a pair baseline unconditionally when replication is enabled late) buffered whole, in memory, on both nodes at once; it is now **bounded at 64 messages** with wait-mode backpressure (unrelated to, and not to be confused with, the rig's separate `MaxBatchSize: 64` DB-read-page pin above), which required moving the snapshot to run alongside the receive loop (not inside the pre-loop handshake) so the inbox is drained for the whole session and can't deadlock itself.
|
||||
- All timestamps are UTC throughout the system.
|
||||
- Inter-cluster communication uses **three** transports, not two — **all cross-cluster command/control and data now rides gRPC** after the ClusterClient→gRPC migration's Phase 4 (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`) deleted Akka `ClusterClient`/`ClusterClientReceptionist`: (1) **gRPC command/control** — site→central over the central-hosted `CentralControlService` (`GrpcCentralTransport`, sticky central-a→central-b channel pair; deployments/notifications/health/heartbeat/audit-ingest/reconcile), and central→site over the site-hosted `SiteCommandService` (`GrpcSiteTransport`, per-site NodeA→NodeB channel pair; the 28 lifecycle/OPC-UA/query/parked/route/failover commands); (2) **gRPC** server-streaming for real-time data (attribute values, alarm states, `SiteStreamService`); and (3) **plain token-gated HTTP** for the deployment config itself — notify-and-fetch, the site pulls the config from `DeploymentConfigEndpoints` (`ManagementService/DeploymentConfigEndpoints.cs`) with an `X-Deployment-Token` header, `AllowAnonymous` with the per-deployment token as the entire security boundary. The gRPC boundary is per-site PSK-authenticated (`ControlPlaneAuthInterceptor`, unchanged). There is **no receptionist registration** — discovery is by dialling configured endpoints; central builds one `SitePairChannelProvider` per site (addresses from `Site.GrpcNodeAAddress`/`GrpcNodeBAddress`, refreshed from the DB every 60s and on admin changes), sites dial `ScadaBridge:Communication:CentralGrpcEndpoints` (both central nodes, h2c on `CentralGrpcPort` 8083, **NOT** via Traefik). **Discovery is asymmetric by design:** central discovers site gRPC addresses from the *database* (refreshable at runtime), sites discover central from *appsettings* (`CentralGrpcEndpoints`, static — restart required; `StartupValidator` requires a Site node to list at least one). `Akka.Cluster.Tools` stays for ClusterSingleton; only the ClusterClient part is gone. **Central never buffers for an unreachable site** — the send fails with the caller's Ask/deadline timing out; a `ConnectionStateChanged` mechanism built for this was deleted as dead code.
|
||||
@@ -141,6 +141,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
|
||||
- Per-instance operation lock covers all mutating commands (deploy, disable, enable, delete).
|
||||
- **Bulk site deployment (`DeploymentService.DeploySiteAsync`, arch-review WP2.5)** redeploys every out-of-date instance at a site through one flatten session (a template chain shared by N instances is walked once) in three phases — serial Prepare, bounded-parallel Send (`SiteDeploymentMaxParallelism`, default 4), serial Finalize — still **not all-or-nothing**: any one instance's failure doesn't block the rest. **Disabled instances are SKIPPED by bulk deploy as of 2026-08-19 (arch-review P7-2)** — they are reported in the summary's additive `SkippedInstances` list, never flattened/sent/re-enabled (single-instance deploy remains the explicit operator path that deploys+enables a Disabled instance); **Up-to-date instances are ALSO skipped as of 2026-08-20 (arch-review P7-6)** — a redeploy is not idempotent at the site (it resets static overrides and clears native alarm state), so `DeploySiteAsync` pre-filters through `IStaleInstanceProbe` (trustworthy since P7-4 fixed its memo timing) and reports unchanged instances in `SkippedInstances` with reason `up-to-date`; the filter is conservative (every uncertainty deploys: missing snapshot, `ShouldQuerySiteBeforeRedeploy`, null/faulting probe). Additive `MgmtDeploySiteCommand.Force` / CLI `deploy site --force` restores the unconditional push for what central cannot see (restored volume, rebuilt node) — **Disabled still wins under `--force`**. Also P7-5: the three phases share one scoped DbContext and a failed `SaveChanges` leaves the entity tracked, so one permanent failure used to cascade to every later instance and strand records `InProgress`; `MarkDeploymentFailedAsync` now detaches pending changes and retries the `Failed` write once on the clean tracker; Prepare's per-instance reconcile also runs under a short `SiteDeploymentReconcileTimeout` (5s) with a once-per-batch unreachable-site circuit-breaker, so a down site no longer costs 30s × N serial before the first send (P7-3). The `PendingDeployment` staging row (5-minute TTL) is written per-instance immediately before that instance's round-trip in the Send phase, not upfront in Prepare, so a large batch's tail instances don't have their TTL clock already half-spent by the time they're reached.
|
||||
- **Site artifact revision + fleet-wide pending-updates indicator (2026-09-14, `docs/plans/2026-09-14-pending-updates-design.md`).** `Site.ArtifactRevisionHash`/`ArtifactsDeployedAt` (migration `AddSiteArtifactRevision`) are stamped by `ArtifactDeploymentService` only on a site whose artifact deploy succeeds, so a redeployed fleet's artifacts are trackable per site the same way instance staleness already is. `PendingUpdatesEvaluator` combines that with `DeploymentService.GetInstanceStalenessAsync` (one shared `FlattenSession` for the whole fleet) into a `PendingUpdatesSnapshot`, held behind a process-wide singleton `IPendingUpdatesCache` (`ScadaBridge:DeploymentManager:PendingUpdatesCacheTtl`, default 60s) so N Blazor circuits cost one fleet flatten per TTL, not one each. Surfaces: the Sites page's per-site **Artifacts** badge (`Current`/`Out of date`/`Never deployed`) + instance-staleness line + header **Re-check**, and a rail-footer **pending updates** indicator (Deployer/Administrator only, 30s poll, site-scoped). **Upgrade consequence:** every site reads *Never deployed* until one artifact deploy runs post-upgrade — deliberate, not a bug. Also shipped: the rail nav now sits in its own scroll pane (`NavMenu`'s `.rail-scroll` + `site.css`) so a long nav can't push the footer/indicator off-screen; the `ZB.MOM.WW.Theme` kit itself is untouched (register row 123 proposes upstreaming it).
|
||||
- **Site secrets ride the artifact deploy; the gRPC secrets hub is retired in ScadaBridge (2026-09-28, `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`).** `Deploy Artifacts` now ships each site **only the secrets it references** — `SiteSecretsCollector` scans a site's data-connection JSON for `secret:<name>` (`SecretReferenceScanner`, protocol-agnostic, bootstrap names like `SB-GRPC-PSK-*` excluded) and resolves each against the central `ISecretStore`; a missing/tombstoned secret fails **that site's** deploy before anything is sent (fail-closed), ciphertext only ever crosses the wire, and secrets factor into `ArtifactRevisionHashService`'s staleness hash (a missing secret hashes as a distinguishable marker, so an unresolvable reference never reads *Current*; a site with no secret references hashes identically whether the field is null or empty). A pre-feature site (answers with no `SecretsApplied`) or a partial apply is also treated as a deploy failure and is never stamped; one `DeploySiteSecrets`/`SiteSecrets` audit row (`secrets` name + revision and `missing` names only, never ciphertext; no `removed` field — a removal-only deploy writes no central row) is written per site deploy that carried secrets or a missing-secret failure, on both the sweep and the per-site retry paths. Site-side: `deployed_secrets` is the 11th LocalDb-replicated table (ciphertext rows only, upsert-then-soft-remove); the per-node `DeployedSecretsMirror` (`SiteRuntime:DeployedSecretsMirrorInterval`, default 15s, also run synchronously right after an active-node apply) reconciles it into that node's local `ISecretStore`, ledgers what it mirrored in the node-local `mirrored_secrets` table so an out-of-band local secret sharing a deployed name is never silently touched (a mirror-owned name deleted or edited locally is healed back to the deployed value on the next pass, even at an unchanged revision), and trial-decrypts every pass to catch a KEK mismatch. Secrets are mirrored before the deploy's data connections reach the DCL; a rotation with unchanged connection config takes effect on the next (re)connect. **A site pair using deployed secrets must run LocalDb replication** — otherwise only the node active at deploy time holds them. Central runs one shared secret store, `Secrets:Store:Provider=SqlServer` (Central role only; a clustered central left on `Sqlite` gets a boot Warning) — the pre-host `${secret:}` expander reads the same store the running node serves. **ALL secrets replication is retired in ScadaBridge** — the pull-only gRPC hub and the SQL-Server replication mode both — `Secrets:Replication:Enabled` set to anything but empty/`false` now fails `StartupValidator` on either role with a message pointing at Deploy Artifacts / `Secrets:Store:Provider=SqlServer`.
|
||||
- **Create Instance picks its template from the Templates folder tree** (2026-08-21): the flat `<select>` is gone. The Template field is a read-only folder-qualified path box (`TemplatePathFormatter`, "Folder / Sub / Name") + a **Choose…** button, both opening `TemplatePickerDialog` — an `IDialogService.ShowAsync` body wrapping the **same shared `TemplateFolderTree`** the Templates page renders, with its own search, expand/collapse-all, templates-only selection and double-click-to-pick. It opens on the current pick via the tree's additive **`RevealKeys`** parameter (ancestors of the named `f:`/`t:` keys start expanded when no `Filter` is active; an active filter's reveal set still wins; callers passing nothing are byte-identical). `InstanceCreate` gained `[SupplyParameterFromQuery] TemplateId`, which the Templates tree's new **Deployer-gated "New Instance…"** context-menu item navigates with; combined with `?siteId=` it populates the Connection Bindings card on first render. **The trap for anyone extending the picker:** in `TreeViewSelectionMode.Single` the inner `TreeView` reads the *singular* `SelectedKey`/`SelectedKeyChanged` pair, but `TemplateFolderTree` forwards only the plural (Checkbox-mode) `SelectedKeys` pair — so a Single-mode caller cannot get a row highlight through the shared tree and must own selection in its own `NodeContent`, as the picker does. Design: `docs/plans/2026-08-21-template-picker-design.md`.
|
||||
- **Create Instance binds module groups at creation** (2026-08-21): the create page's Connection Bindings card flattens template+site through the additive `IFlatteningPipeline.PreviewAsync` (an unsaved instance — no validation, no hash, nothing persisted, instance repo never touched) and reuses `BindingGroupBuilder`; `NodeBrowserDialog.SelectableNodeClasses` lets it pick **Object** nodes (default stays Variable-only for every existing caller). Create = `CreateInstanceAsync` → `SetConnectionBindingsAsync` (the existing upsert, no new server surface) → always navigate to Configure with `?created=1&bound=N|&bindError=…`, because the row exists whether or not the second call succeeded. Partial binding is legal: a group missing a connection or an Object is skipped and stays unbound. Design: `docs/plans/2026-08-21-create-instance-bindings-design.md`.
|
||||
- Site-side apply is all-or-nothing per instance.
|
||||
@@ -174,7 +175,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
|
||||
- One row per lifecycle event; cached calls produce 4+ rows per operation (`Submitted`, `Forwarded`, `Attempted`, `Delivered`/`Parked`/`Discarded`).
|
||||
- `ExecutionId` (`uniqueidentifier NULL`) is the universal per-run correlation value — every audit row emitted by one script execution / inbound request shares it; `CorrelationId` remains the per-operation lifecycle id (NULL for sync one-shots).
|
||||
- `ParentExecutionId` (`uniqueidentifier NULL`) is the cross-execution spawn pointer — every row of a spawned run carries the spawner's `ExecutionId`; bridges inbound API → routed-site-script, alarm-triggered on-trigger scripts, and nested `CallScript`/`CallShared` invocations; `IX_AuditLog_ParentExecution` backs the filter + the recursive execution-tree walk. **Tag-cascade (alarm leg) is populated, not just plumbed** — M5.4 T4 threaded the `parentExecutionId` parameter but every `AlarmActor.SpawnAlarmExecution` call site passed null, so alarm runs were silently always roots; the id now rides site-locally as `SetStaticAttributeCommand.SourceExecutionId` → `AttributeValueChanged.SourceExecutionId` → `SpawnAlarmExecution` (all additive/nullable, no wire/proto/schema change; `Expression` triggers capture the writer *with* the evaluated snapshot since the eval completes off-dispatcher). Sources are `ScriptRuntimeContext.SetAttribute` (the run's own `ExecutionId`) and `Route.To(...).SetAttributes(...)` (the inbound request's). **Still roots by design, not omission:** alarms fired by DCL data (external values have no spawning execution — including the device echo of a script write to a *data-sourced* attribute, so only **static** writes cascade), and `ScriptActor` value-change/conditional/expression/timer trigger runs (a timer tick has no spawner; a `WhileTrue`/interval run has no single identifiable write).
|
||||
- Site SQLite hot-path first, then gRPC telemetry to central; ingest is idempotent on `EventId`; periodic reconciliation pull as fallback when telemetry is lost. **The site audit DB is a separate, unreplicated SQLite file — NOT one of LocalDb's ten consolidated tables** — at the **required** `AuditLog:SiteWriter:DatabasePath` (arch-review WP1.2, finding #2 High), `/app/data/auditlog.db` on the rig; `StartupValidator` refuses to boot a Site node without it. `DatabasePath` used to default to CWD-relative `auditlog.db`, which on the docker rig landed on the container's ephemeral overlay, not the mounted volume — the same data-loss pattern the pre-LocalDb legacy S&F/tracking databases hit (see the LocalDb bullet above), independently rediscovered here because this file sits outside LocalDb's management. The writer loop now also honors `FlushIntervalMs` (was validated but never read — one fsync'ing commit per event even at trickle rate) and sets `PRAGMA synchronous = NORMAL` (audit is best-effort by design, so the narrower power-loss window is an acceptable trade for far fewer fsyncs; WAL still guarantees no corruption).
|
||||
- Site SQLite hot-path first, then gRPC telemetry to central; ingest is idempotent on `EventId`; periodic reconciliation pull as fallback when telemetry is lost. **The site audit DB is a separate, unreplicated SQLite file — NOT one of LocalDb's eleven consolidated tables** — at the **required** `AuditLog:SiteWriter:DatabasePath` (arch-review WP1.2, finding #2 High), `/app/data/auditlog.db` on the rig; `StartupValidator` refuses to boot a Site node without it. `DatabasePath` used to default to CWD-relative `auditlog.db`, which on the docker rig landed on the container's ephemeral overlay, not the mounted volume — the same data-loss pattern the pre-LocalDb legacy S&F/tracking databases hit (see the LocalDb bullet above), independently rediscovered here because this file sits outside LocalDb's management. The writer loop now also honors `FlushIntervalMs` (was validated but never read — one fsync'ing commit per event even at trickle rate) and sets `PRAGMA synchronous = NORMAL` (audit is best-effort by design, so the narrower power-loss window is an acceptable trade for far fewer fsyncs; WAL still guarantees no corruption).
|
||||
- Cached operations: site emits a single additively-extended `CachedCallTelemetry` packet carrying both audit events and operational state; central writes `AuditLog` + `SiteCalls` in one transaction.
|
||||
- Payload cap 8 KB by default / 64 KB on error rows; auth headers redacted by default; SQL parameter values captured by default; per-target redaction opt-in. Inbound API: full verbatim capture up to `InboundMaxBytes` (default 1 MiB); request headers stored in `Extra.requestHeaders` (post-redaction); per-method `SkipBodyCapture` flag suppresses bodies while still recording headers + metadata; `AuditInboundCeilingHits` counter surfaced on health snapshot. (M5.3 T7)
|
||||
- Audit-write failure NEVER aborts the user-facing action — audit is best-effort, the action's own success/failure path is authoritative.
|
||||
|
||||
@@ -109,7 +109,6 @@
|
||||
<PackageVersion Include="ZB.MOM.WW.Secrets.Abstractions" Version="0.6.2" />
|
||||
<PackageVersion Include="ZB.MOM.WW.Secrets.Ui" Version="0.6.2" />
|
||||
<PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.SqlServer" Version="0.6.2" />
|
||||
<PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.Grpc" Version="0.6.2" />
|
||||
<PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.3.4" />
|
||||
<PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.3.4" />
|
||||
<PackageVersion Include="ZB.MOM.WW.LocalDb.Contracts" Version="0.3.4" />
|
||||
|
||||
+71
-65
@@ -17,59 +17,53 @@
|
||||
x-secrets-store-env: &secrets-store-env
|
||||
Secrets__SqlitePath: "/data/scadabridge-secrets.db"
|
||||
|
||||
# ── Clustered secret replication: the pull-only gRPC hub (scadaproj#3) ─────────
|
||||
# ── Secrets:Store:Provider=SqlServer: central shares one secret store directly (Task 5) ──
|
||||
#
|
||||
# Central hosts the hub on its EXISTING h2c control-plane listener (CentralGrpcPort
|
||||
# 8083, alongside CentralControlService); each site node sweeps it on an interval and
|
||||
# writes what it pulls into its OWN local SQLite store. Nothing but ciphertext crosses
|
||||
# the wire, so every participating node must resolve the SAME KEK.
|
||||
# Task 11 retired secrets replication entirely — both the pull-only gRPC hub (central
|
||||
# hosted, sites swept it) and the SQL-Server ReplicatingSecretStore mode that preceded it.
|
||||
# Sites now receive secrets exclusively via Deploy Artifacts; there is no hub and nothing
|
||||
# for a site to sweep. What remains, and is unrelated to replication, is Secrets:Store:
|
||||
# Provider=SqlServer: it points BOTH central nodes at ONE shared SQL-Server database
|
||||
# directly — no hub, no follower sweep, just two nodes reading and writing the same rows —
|
||||
# so a clustered central pair can never serve sites a drifted secret set. central-a/b set
|
||||
# it explicitly below; site nodes always ignore the key and keep their local SQLite store.
|
||||
#
|
||||
# ENABLED ON FOUR NODES ONLY: the central pair (hub) and the site-a pair (followers).
|
||||
# site-b and site-c are deliberately left without it, so the default-OFF posture is
|
||||
# proven side by side on one rig — exactly as site-a is the rig's only LocalDb-replicated
|
||||
# pair. A node with no Secrets__* override keeps the shipped appsettings default
|
||||
# (Replication:Enabled=false, Mode=SqlServer) and composes a plain local store.
|
||||
#
|
||||
# ALL VALUES HERE ARE DEV-ONLY and committed under the same exception as the mesh PSKs
|
||||
# and the ApiKeyPepper above: a local docker rig needs a working credential in source
|
||||
# control to boot. Production supplies the KEK out of band (ZB_SECRETS_MASTER_KEY, never
|
||||
# committed) and the hub token from appsettings/env — NEVER as a ${secret:} reference,
|
||||
# since resolving one is what the hub exists to make possible.
|
||||
x-secrets-hub-env: &secrets-hub-env
|
||||
# DEV-ONLY KEK — NOT a real key. Identical on all four participating nodes: only
|
||||
# ciphertext replicates, so a node with a different KEK fails closed on resolve with a
|
||||
# kek_id mismatch that reads like corruption but is a deployment error.
|
||||
ZB_SECRETS_MASTER_KEY: "zZiBWuoaVMbJmGXToLk9Lakw0iJozXoL/7Gxac3GwJ4="
|
||||
Secrets__Replication__Enabled: "true"
|
||||
Secrets__Replication__Mode: "Grpc"
|
||||
# DEV-ONLY shared bearer token — NOT a real secret. Presented by every follower and
|
||||
# verified by the hub's fail-closed SecretsHubAuthInterceptor. Must be IDENTICAL on the
|
||||
# hub and every follower; an unset token is a startup failure on both halves.
|
||||
Secrets__GrpcHub__BearerToken: "secrets-hub-docker-dev-token"
|
||||
|
||||
# Central-only half: the SHARED SQL-Server secret store (scadaproj#4, Secrets 0.5.0).
|
||||
# In Grpc mode BOTH central nodes read and write ONE copy of every row in this database,
|
||||
# so the two hub instances serve identical manifests by construction — that is what makes
|
||||
# the site-side FallbackEndpoints below safe. ZbSecretsHub is a dedicated database on the
|
||||
# rig's existing scadabridge-mssql container (default zbsecrets schema, created by the
|
||||
# boot-time SqlServerSecretsStoreMigrator; the database itself + the scadabridge_app grant
|
||||
# were provisioned once via sqlcmd — see docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md).
|
||||
# Same DEV-ONLY credentials as the ScadaBridge__Database__* strings above. Must be a
|
||||
# LITERAL value, never a ${secret:} reference — the pre-host expander needs this string to
|
||||
# reach the store that would resolve it (bootstrap circularity; registration rejects it).
|
||||
# Site nodes must NEVER carry this key: sites talk to central, not to central's database.
|
||||
x-secrets-hub-central-env: &secrets-hub-central-env
|
||||
# ZbSecretsHub is a dedicated database on the rig's existing scadabridge-mssql container
|
||||
# (default zbsecrets schema, created by the boot-time SqlServerSecretsStoreMigrator; the
|
||||
# database itself + the scadabridge_app grant were provisioned once via sqlcmd — see
|
||||
# docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md). Same DEV-ONLY
|
||||
# credentials as the ScadaBridge__Database__* strings below. Must be a LITERAL value,
|
||||
# never a ${secret:} reference — the pre-host expander needs this string to reach the
|
||||
# store that would resolve it (bootstrap circularity; registration rejects it). Site nodes
|
||||
# must NEVER carry this key: sites talk to central, not to central's database.
|
||||
x-secrets-central-store-env: &secrets-central-store-env
|
||||
Secrets__Store__Provider: "SqlServer"
|
||||
Secrets__SqlServer__ConnectionString: "Server=scadabridge-mssql,1433;Database=ZbSecretsHub;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
|
||||
|
||||
# Site half of the same section. The hub client dials Endpoint first and fails over, per
|
||||
# call, to FallbackEndpoints in order (sticky on whichever answered last, Secrets 0.5.0).
|
||||
# Listing central-b is safe ONLY because both centrals serve the one shared SQL store
|
||||
# above — never list endpoints backed by independent stores: failing over to an emptier
|
||||
# hub is a silent convergence stop, the exact defect (scadaproj#4) the shared store
|
||||
# exists to prevent.
|
||||
x-secrets-hub-site-env: &secrets-hub-site-env
|
||||
Secrets__GrpcHub__Endpoint: "http://scadabridge-central-a:8083"
|
||||
Secrets__GrpcHub__FallbackEndpoints__0: "http://scadabridge-central-b:8083"
|
||||
# ── Deploy-artifact secret KEKs ──────────────────────────────────────────────────────
|
||||
#
|
||||
# Every node that HANDLES a deployed secret — central (it encrypts what ships in the
|
||||
# artifact) and site-a (it decrypts what it receives) — must resolve the SAME
|
||||
# ZB_SECRETS_MASTER_KEY: only ciphertext ever crosses the wire, so a node with a different
|
||||
# KEK fails closed on resolve with a kek_id mismatch that reads like corruption but is a
|
||||
# deployment error. site-b/site-c deliberately carry a DIFFERENT dev KEK (below) so that
|
||||
# mismatch is a real, provable negative on this rig rather than an assumption — see
|
||||
# Task 13.
|
||||
#
|
||||
# ALL VALUES HERE ARE DEV-ONLY and committed under the same exception as the mesh PSKs
|
||||
# and the ApiKeyPepper below: a local docker rig needs a working credential in source
|
||||
# control to boot. Production supplies the KEK out of band (ZB_SECRETS_MASTER_KEY, never
|
||||
# committed).
|
||||
x-secrets-shared-kek-env: &secrets-shared-kek-env
|
||||
# DEV-ONLY KEK — NOT a real key. Identical on central-a/b and site-a-a/b.
|
||||
ZB_SECRETS_MASTER_KEY: "zZiBWuoaVMbJmGXToLk9Lakw0iJozXoL/7Gxac3GwJ4="
|
||||
|
||||
# A DIFFERENT DEV-ONLY KEK — NOT a real key — deliberately mismatched from the one above.
|
||||
# site-b and site-c never receive a deployed secret from this rig's exercised paths today,
|
||||
# but carrying a mismatched key here (rather than none) means a future deploy targeting
|
||||
# them fails closed with a named kek_id mismatch instead of an untested code path.
|
||||
x-secrets-mismatched-kek-env: &secrets-mismatched-kek-env
|
||||
ZB_SECRETS_MASTER_KEY: "esKrmqV2RBmMq/+6T/5zK2BcalRy5gzDXhoG9Qydg6Q="
|
||||
|
||||
services:
|
||||
central-a:
|
||||
@@ -87,9 +81,9 @@ services:
|
||||
stop_grace_period: 30s
|
||||
container_name: scadabridge-central-a
|
||||
environment:
|
||||
# Hub half of the pull-only gRPC secrets hub + the central-only shared SQL store
|
||||
# (anchors at the top of this file).
|
||||
<<: [*secrets-store-env, *secrets-hub-env, *secrets-hub-central-env]
|
||||
# Central-only shared SQL secret store (Secrets:Store:Provider=SqlServer, Task 5) + the
|
||||
# deploy-artifact secret KEK (anchors at the top of this file).
|
||||
<<: [*secrets-store-env, *secrets-shared-kek-env, *secrets-central-store-env]
|
||||
SCADABRIDGE_CONFIG: Central
|
||||
ASPNETCORE_ENVIRONMENT: Development
|
||||
ASPNETCORE_URLS: "http://+:5000"
|
||||
@@ -154,9 +148,9 @@ services:
|
||||
stop_grace_period: 30s
|
||||
container_name: scadabridge-central-b
|
||||
environment:
|
||||
# Hub half of the pull-only gRPC secrets hub + the central-only shared SQL store
|
||||
# (anchors at the top of this file).
|
||||
<<: [*secrets-store-env, *secrets-hub-env, *secrets-hub-central-env]
|
||||
# Central-only shared SQL secret store (Secrets:Store:Provider=SqlServer, Task 5) + the
|
||||
# deploy-artifact secret KEK (anchors at the top of this file).
|
||||
<<: [*secrets-store-env, *secrets-shared-kek-env, *secrets-central-store-env]
|
||||
SCADABRIDGE_CONFIG: Central
|
||||
ASPNETCORE_ENVIRONMENT: Development
|
||||
ASPNETCORE_URLS: "http://+:5000"
|
||||
@@ -221,9 +215,11 @@ services:
|
||||
stop_grace_period: 30s
|
||||
container_name: scadabridge-site-a-a
|
||||
environment:
|
||||
# Follower half of the pull-only gRPC secrets hub (anchors at the top of this
|
||||
# file). site-b and site-c deliberately carry neither.
|
||||
<<: [*secrets-store-env, *secrets-hub-env, *secrets-hub-site-env]
|
||||
# The shared deploy-artifact secret KEK (anchors at the top of this file) — this node
|
||||
# must decrypt what central ships in a deploy artifact. site-b and site-c deliberately
|
||||
# carry a DIFFERENT (mismatched) KEK instead, so that failure mode is provable on this
|
||||
# rig rather than assumed (Task 13).
|
||||
<<: [*secrets-store-env, *secrets-shared-kek-env]
|
||||
SCADABRIDGE_CONFIG: Site
|
||||
# Register row 63. The central containers have always set this; the site containers
|
||||
# never did, so the Development-only DI scope validation (register row 50,
|
||||
@@ -258,9 +254,11 @@ services:
|
||||
stop_grace_period: 30s
|
||||
container_name: scadabridge-site-a-b
|
||||
environment:
|
||||
# Follower half of the pull-only gRPC secrets hub (anchors at the top of this
|
||||
# file). site-b and site-c deliberately carry neither.
|
||||
<<: [*secrets-store-env, *secrets-hub-env, *secrets-hub-site-env]
|
||||
# The shared deploy-artifact secret KEK (anchors at the top of this file) — this node
|
||||
# must decrypt what central ships in a deploy artifact. site-b and site-c deliberately
|
||||
# carry a DIFFERENT (mismatched) KEK instead, so that failure mode is provable on this
|
||||
# rig rather than assumed (Task 13).
|
||||
<<: [*secrets-store-env, *secrets-shared-kek-env]
|
||||
SCADABRIDGE_CONFIG: Site
|
||||
# Register row 63. The central containers have always set this; the site containers
|
||||
# never did, so the Development-only DI scope validation (register row 50,
|
||||
@@ -295,7 +293,9 @@ services:
|
||||
stop_grace_period: 30s
|
||||
container_name: scadabridge-site-b-a
|
||||
environment:
|
||||
<<: *secrets-store-env
|
||||
# A DIFFERENT (mismatched) deploy-artifact secret KEK from central/site-a's — see the
|
||||
# anchor comment at the top of this file (Task 13's mismatch negative).
|
||||
<<: [*secrets-store-env, *secrets-mismatched-kek-env]
|
||||
SCADABRIDGE_CONFIG: Site
|
||||
# Register row 63. The central containers have always set this; the site containers
|
||||
# never did, so the Development-only DI scope validation (register row 50,
|
||||
@@ -329,7 +329,9 @@ services:
|
||||
stop_grace_period: 30s
|
||||
container_name: scadabridge-site-b-b
|
||||
environment:
|
||||
<<: *secrets-store-env
|
||||
# A DIFFERENT (mismatched) deploy-artifact secret KEK from central/site-a's — see the
|
||||
# anchor comment at the top of this file (Task 13's mismatch negative).
|
||||
<<: [*secrets-store-env, *secrets-mismatched-kek-env]
|
||||
SCADABRIDGE_CONFIG: Site
|
||||
# Register row 63. The central containers have always set this; the site containers
|
||||
# never did, so the Development-only DI scope validation (register row 50,
|
||||
@@ -363,7 +365,9 @@ services:
|
||||
stop_grace_period: 30s
|
||||
container_name: scadabridge-site-c-a
|
||||
environment:
|
||||
<<: *secrets-store-env
|
||||
# A DIFFERENT (mismatched) deploy-artifact secret KEK from central/site-a's — see the
|
||||
# anchor comment at the top of this file (Task 13's mismatch negative).
|
||||
<<: [*secrets-store-env, *secrets-mismatched-kek-env]
|
||||
SCADABRIDGE_CONFIG: Site
|
||||
# Register row 63. The central containers have always set this; the site containers
|
||||
# never did, so the Development-only DI scope validation (register row 50,
|
||||
@@ -397,7 +401,9 @@ services:
|
||||
stop_grace_period: 30s
|
||||
container_name: scadabridge-site-c-b
|
||||
environment:
|
||||
<<: *secrets-store-env
|
||||
# A DIFFERENT (mismatched) deploy-artifact secret KEK from central/site-a's — see the
|
||||
# anchor comment at the top of this file (Task 13's mismatch negative).
|
||||
<<: [*secrets-store-env, *secrets-mismatched-kek-env]
|
||||
SCADABRIDGE_CONFIG: Site
|
||||
# Register row 63. The central containers have always set this; the site containers
|
||||
# never did, so the Development-only DI scope validation (register row 50,
|
||||
|
||||
Executable
+593
@@ -0,0 +1,593 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
# Live gate for SITE SECRETS RIDING THE ARTIFACT DEPLOY (design
|
||||
# docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md, plan task 13). Drives the real
|
||||
# central paths end to end on the running docker rig:
|
||||
#
|
||||
# - secrets are created / rotated / deleted at central through the Central UI's own
|
||||
# /admin/secrets page (ZB.MOM.WW.Secrets.Ui over the central pair's shared SQL store) — the
|
||||
# supported operator path; there is no CLI/management verb for central secrets
|
||||
# - data connections and artifact deploys go through the ScadaBridge CLI
|
||||
# (CLI -> Traefik -> active central -> ArtifactDeploymentService)
|
||||
# - the Artifacts badge is read from the Sites page after pressing Re-check (the pending-updates
|
||||
# evaluation has no CLI/management surface; it is Blazor-interactive, so a headless browser is
|
||||
# the only honest reader)
|
||||
# - each site node's own secret store (the library's SQLite `secret` table in
|
||||
# /app/data/scadabridge-secrets.db) and LocalDb (`deployed_secrets`, `mirrored_secrets`,
|
||||
# `data_connection_definitions` in /app/data/site-localdb.db) are copied out with `docker cp`
|
||||
# (the image carries no sqlite3) and read with the host's python3 sqlite3 module
|
||||
# - the central row is read (never written) from ZbSecretsHub.zbsecrets.secret with sqlcmd
|
||||
#
|
||||
# Ciphertext is compared by SHA-256 only; secret VALUES are random throwaways held in shell
|
||||
# variables and handed to the browser through the environment — never printed, never on argv.
|
||||
#
|
||||
# Checks (each prints PASS/FAIL; the run ends "PASS n/8" and exits non-zero on any FAIL):
|
||||
# 1 secret + secret-referencing MxGateway connection on site-a -> site-a Out of date, b/c Current
|
||||
# 2 deploy site-a -> success, secretsDeployed 1, site-a Current
|
||||
# 3 both site-a nodes hold the secret at the central revision + ciphertext (standby <= 30 s)
|
||||
# 4 rotate at central -> site-a Out of date -> deploy -> both nodes at the new revision
|
||||
# 5 drop the secret: reference -> deploy -> both nodes no longer hold it; an out-of-band secret
|
||||
# seeded on site-a-a through the library's own write path (the ZB.MOM.WW.Secrets `secret` CLI
|
||||
# run against that node's store) survives
|
||||
# 6 reference a non-existent secret -> site-a deploy fails closed with the named message,
|
||||
# site-a stays Out of date, nothing changed on either site-a node
|
||||
# 7 the same reference on site-b (deliberately mismatched dev KEK) -> deploy fails naming the KekId
|
||||
# 8 one DeploySiteSecrets audit row per secret-bearing deploy; no plaintext / base64 ciphertext
|
||||
# in any of the 8 nodes' logs
|
||||
#
|
||||
# Teardown (trap EXIT, also run up-front so a rerun starts clean): gate connections deleted,
|
||||
# site-a + site-b redeployed so the removal reaches them, the central gate secret deleted, the
|
||||
# out-of-band secret removed from site-a-a, then a fleet-wide artifact deploy so the rig ends with
|
||||
# every site Current. The secret library tombstones rather than hard-deletes, so deleted gate
|
||||
# names remain as is_deleted=1 rows in the stores — that is the library's contract, not residue.
|
||||
#
|
||||
# Prereqs: rig up (8 nodes ready), `python3 -m playwright` with chromium, the ScadaBridge CLI
|
||||
# built, and scadaproj checked out next to this repo (the `secret` CLI is published from it once
|
||||
# into $SECRETS_CLI_DIR).
|
||||
#
|
||||
# Usage: bash docker/verify-site-secrets.sh
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
CLI="${SCADABRIDGE_CLI:-$PROJECT_ROOT/src/ZB.MOM.WW.ScadaBridge.CLI/bin/Debug/net10.0/scadabridge}"
|
||||
URL="${SCADABRIDGE_URL:-http://localhost:9000}"
|
||||
A=(--url "$URL" --format=json --username "${SCADABRIDGE_USERNAME:-multi-role}" --password "${SCADABRIDGE_PASSWORD:-password}")
|
||||
SECRETS_CLI_PROJECT="${SECRETS_CLI_PROJECT:-$HOME/Desktop/scadaproj/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Cli}"
|
||||
SECRETS_CLI_DIR="${SECRETS_CLI_DIR:-${TMPDIR:-/tmp}/sb-gate-secrets-cli}"
|
||||
STANDBY_WINDOW_S="${STANDBY_WINDOW_S:-30}"
|
||||
|
||||
# DEV-ONLY rig KEK shared by central + site-a (docker/docker-compose.yml, x-secrets-shared-kek-env).
|
||||
SITE_A_KEK="zZiBWuoaVMbJmGXToLk9Lakw0iJozXoL/7Gxac3GwJ4="
|
||||
MSSQL_CONTAINER="${MSSQL_CONTAINER:-scadabridge-mssql}"
|
||||
MSSQL_USER="scadabridge_app"; MSSQL_PASS='ScadaBridge_Dev1#'
|
||||
|
||||
SECRET="sb-gate-mxgw"
|
||||
MISSING="sb-gate-missing"
|
||||
OOB="sb-gate-oob"
|
||||
CONN_A="sb-gate-conn-a"
|
||||
CONN_B="sb-gate-conn-b"
|
||||
ACTIVE_A="scadabridge-site-a-a"; STANDBY_A="scadabridge-site-a-b"
|
||||
NODES=(central-a central-b site-a-a site-a-b site-b-a site-b-b site-c-a site-c-b)
|
||||
|
||||
VALUE_1=$(python3 -c 'import secrets; print("gate1-" + secrets.token_hex(16))')
|
||||
VALUE_2=$(python3 -c 'import secrets; print("gate2-" + secrets.token_hex(16))')
|
||||
VALUE_OOB=$(python3 -c 'import secrets; print("gateoob-" + secrets.token_hex(16))')
|
||||
|
||||
CHECKS_PASSED=0; CHECKS_FAILED=0
|
||||
CHECK_OK=1
|
||||
begin_check() { CHECK_OK=1; echo ""; echo "=== Check $1: $2 ==="; }
|
||||
ok() { echo " ok $*"; }
|
||||
bad() { CHECK_OK=0; echo " FAIL $*"; }
|
||||
end_check() {
|
||||
if [ "$CHECK_OK" -eq 1 ]; then CHECKS_PASSED=$((CHECKS_PASSED+1)); echo " PASS check $1"
|
||||
else CHECKS_FAILED=$((CHECKS_FAILED+1)); echo " FAIL check $1"; fi
|
||||
}
|
||||
|
||||
[ -x "$CLI" ] || { echo "CLI not found at $CLI — build it: dotnet build src/ZB.MOM.WW.ScadaBridge.CLI"; exit 1; }
|
||||
python3 -c 'import playwright.sync_api' 2>/dev/null || { echo "python3 playwright missing (pip install playwright && playwright install chromium)"; exit 1; }
|
||||
|
||||
WORK=$(mktemp -d "${TMPDIR:-/tmp}/sb-gate-site-secrets.XXXXXX")
|
||||
HELPER="$WORK/gate.py"
|
||||
|
||||
# ── python helper: browser (UI) + read-only store readers ──────────────────────────────────────
|
||||
cat >"$HELPER" <<'PY'
|
||||
import hashlib, json, os, sqlite3, subprocess, sys, tempfile, time, shutil
|
||||
|
||||
def sha(b):
|
||||
return hashlib.sha256(b if isinstance(b, (bytes, bytearray)) else b.encode()).hexdigest()
|
||||
|
||||
# ---- Central UI -------------------------------------------------------------------------------
|
||||
def _browser(fn):
|
||||
from playwright.sync_api import sync_playwright
|
||||
with sync_playwright() as p:
|
||||
b = p.chromium.launch()
|
||||
try:
|
||||
pg = b.new_page()
|
||||
return fn(pg)
|
||||
finally:
|
||||
b.close()
|
||||
|
||||
def _wait_circuit(pg):
|
||||
# Blazor Server: handlers only work once the circuit is up.
|
||||
pg.wait_for_function("() => window.Blazor !== undefined", timeout=30000)
|
||||
time.sleep(2)
|
||||
|
||||
def badges(url):
|
||||
def run(pg):
|
||||
pg.goto(url + "/admin/sites")
|
||||
pg.wait_for_selector("button[aria-label='Re-check pending updates']", timeout=30000)
|
||||
_wait_circuit(pg)
|
||||
pg.click("button[aria-label='Re-check pending updates']")
|
||||
time.sleep(1)
|
||||
pg.wait_for_function("() => { const b=document.querySelector(\"button[aria-label='Re-check pending updates']\"); return b && !b.disabled; }", timeout=120000)
|
||||
pg.wait_for_function("() => !document.querySelector('[aria-label=\"Artifacts: checking\"]')", timeout=60000)
|
||||
out = {}
|
||||
for card in pg.query_selector_all(".card"):
|
||||
code = card.query_selector("code"); badge = card.query_selector("[aria-label^='Artifacts:']")
|
||||
if code and badge:
|
||||
out[code.inner_text().strip()] = badge.get_attribute("aria-label").split(": ", 1)[1]
|
||||
return out
|
||||
print(json.dumps(_browser(run)))
|
||||
|
||||
def _row_exists(pg, name):
|
||||
return pg.query_selector(f"button.rotate-secret[data-secret-name='{name}']") is not None
|
||||
|
||||
def _secrets_page(pg, url):
|
||||
pg.goto(url + "/admin/secrets")
|
||||
pg.wait_for_selector("[data-testid='add-secret']", timeout=30000)
|
||||
pg.wait_for_function("() => !document.body.innerText.includes('Loading…')", timeout=30000)
|
||||
_wait_circuit(pg)
|
||||
|
||||
def secret_upsert(url, name, desc):
|
||||
value = os.environ["SB_GATE_VALUE"]
|
||||
def run(pg):
|
||||
_secrets_page(pg, url)
|
||||
mode = "rotate" if _row_exists(pg, name) else "add"
|
||||
for attempt in range(5):
|
||||
if mode == "rotate":
|
||||
pg.click(f"button.rotate-secret[data-secret-name='{name}']")
|
||||
else:
|
||||
pg.click("[data-testid='add-secret']")
|
||||
try:
|
||||
pg.wait_for_selector("[data-testid='editor-submit']", timeout=3000); break
|
||||
except Exception:
|
||||
time.sleep(1)
|
||||
if mode == "add":
|
||||
pg.fill("[data-testid='secret-name']", name); pg.press("[data-testid='secret-name']", "Tab")
|
||||
pg.fill("[data-testid='secret-description']", desc); pg.press("[data-testid='secret-description']", "Tab")
|
||||
pg.fill("[data-testid='secret-value']", value); pg.press("[data-testid='secret-value']", "Tab")
|
||||
pg.click("[data-testid='editor-submit']")
|
||||
deadline = time.time() + 20
|
||||
while time.time() < deadline:
|
||||
err = pg.query_selector("[data-testid='editor-error']")
|
||||
if err:
|
||||
return "error: " + err.inner_text()
|
||||
if pg.query_selector("[data-testid='editor-submit']") is None and _row_exists(pg, name):
|
||||
return mode
|
||||
time.sleep(0.3)
|
||||
return "error: editor did not close"
|
||||
print(_browser(run))
|
||||
|
||||
def secret_delete(url, name):
|
||||
def run(pg):
|
||||
_secrets_page(pg, url)
|
||||
if not _row_exists(pg, name):
|
||||
return "absent"
|
||||
for attempt in range(5):
|
||||
pg.click(f"button.delete-secret[data-secret-name='{name}']")
|
||||
try:
|
||||
pg.wait_for_selector("[data-testid='confirm-delete']", timeout=3000); break
|
||||
except Exception:
|
||||
time.sleep(1)
|
||||
pg.click("[data-testid='confirm-delete']")
|
||||
deadline = time.time() + 20
|
||||
while time.time() < deadline:
|
||||
err = pg.query_selector("[data-testid='delete-error']")
|
||||
if err:
|
||||
return "error: " + err.inner_text()
|
||||
if not _row_exists(pg, name):
|
||||
return "deleted"
|
||||
time.sleep(0.3)
|
||||
return "error: row still listed"
|
||||
print(_browser(run))
|
||||
|
||||
# ---- read-only store readers ------------------------------------------------------------------
|
||||
def _copy_db(container, path):
|
||||
"""docker cp the db + its WAL (NOT the -shm: SQLite rebuilds the wal-index from the WAL on
|
||||
open, which is safer than trusting a copied live shared-memory file)."""
|
||||
d = tempfile.mkdtemp(prefix="sbgate-")
|
||||
base = os.path.basename(path)
|
||||
for suffix in ("-wal", ""):
|
||||
r = subprocess.run(["docker", "cp", f"{container}:{path}{suffix}", os.path.join(d, base + suffix)],
|
||||
capture_output=True)
|
||||
if r.returncode != 0 and suffix == "":
|
||||
raise SystemExit(f"docker cp {container}:{path} failed: {r.stderr.decode().strip()}")
|
||||
return d, os.path.join(d, base)
|
||||
|
||||
def _query(container, path, sql):
|
||||
# Two consecutive reads must agree — guards against a checkpoint racing the two docker cp's.
|
||||
last = None
|
||||
for _ in range(4):
|
||||
d, db = _copy_db(container, path)
|
||||
try:
|
||||
con = sqlite3.connect(db); rows = con.execute(sql).fetchall(); con.close()
|
||||
finally:
|
||||
shutil.rmtree(d, ignore_errors=True)
|
||||
if rows == last:
|
||||
return rows
|
||||
last = rows
|
||||
return last
|
||||
|
||||
def site_secrets(container):
|
||||
rows = _query(container, "/app/data/scadabridge-secrets.db",
|
||||
"SELECT name, revision, kek_id, is_deleted, ciphertext FROM secret WHERE name LIKE 'sb-gate-%' ORDER BY name")
|
||||
print(json.dumps({n: {"rev": r, "kek": k, "deleted": bool(dl), "sha": sha(c)} for n, r, k, dl, c in rows}, sort_keys=True))
|
||||
|
||||
def site_localdb(container):
|
||||
db = "/app/data/site-localdb.db"
|
||||
dep = _query(container, db, "SELECT name, revision, is_removed, row_json FROM deployed_secrets WHERE name LIKE 'sb-gate-%' ORDER BY name")
|
||||
mir = _query(container, db, "SELECT name, revision FROM mirrored_secrets WHERE name LIKE 'sb-gate-%' ORDER BY name")
|
||||
dcs = _query(container, db, "SELECT name, configuration FROM data_connection_definitions WHERE name LIKE 'sb-gate-%' ORDER BY name")
|
||||
print(json.dumps({
|
||||
"deployed": {n: {"rev": r, "removed": bool(x), "sha": sha(j)} for n, r, x, j in dep},
|
||||
"mirrored": {n: r for n, r in mir},
|
||||
"connections": {n: sha(c or "") for n, c in dcs},
|
||||
}, sort_keys=True))
|
||||
|
||||
def central_secret(name):
|
||||
q = ("SET NOCOUNT ON; SELECT name, revision, kek_id, is_deleted, "
|
||||
"LOWER(CONVERT(varchar(64), HASHBYTES('SHA2_256', ciphertext), 2)), "
|
||||
"CONVERT(varchar(max), ciphertext, 2) "
|
||||
f"FROM zbsecrets.secret WHERE name = N'{name}'")
|
||||
r = subprocess.run(["docker", "exec", os.environ["MSSQL_CONTAINER"], "/opt/mssql-tools18/bin/sqlcmd",
|
||||
"-S", "localhost", "-U", os.environ["MSSQL_USER"], "-P", os.environ["MSSQL_PASS"],
|
||||
"-C", "-d", "ZbSecretsHub", "-h", "-1", "-W", "-s", "|", "-Q", q],
|
||||
capture_output=True, text=True)
|
||||
if r.returncode != 0:
|
||||
raise SystemExit("sqlcmd failed: " + r.stderr.strip())
|
||||
lines = [l for l in r.stdout.splitlines() if l.strip()]
|
||||
if not lines:
|
||||
print(json.dumps(None)); return
|
||||
n, rev, kek, dl, h, hexct = lines[0].split("|")
|
||||
import base64
|
||||
b64 = base64.b64encode(bytes.fromhex(hexct)).decode()
|
||||
if sha(bytes.fromhex(hexct)) != h:
|
||||
raise SystemExit("ciphertext hex truncated by sqlcmd")
|
||||
# b64 is only used by check 8's log grep; it is written to a file by the caller, never printed.
|
||||
print(json.dumps({"rev": int(rev), "kek": kek, "deleted": dl == "1", "sha": h, "b64": b64}))
|
||||
|
||||
cmd, args = sys.argv[1], sys.argv[2:]
|
||||
{"badges": lambda: badges(*args),
|
||||
"secret-upsert": lambda: secret_upsert(*args),
|
||||
"secret-delete": lambda: secret_delete(*args),
|
||||
"site-secrets": lambda: site_secrets(*args),
|
||||
"site-localdb": lambda: site_localdb(*args),
|
||||
"central-secret": lambda: central_secret(*args)}[cmd]()
|
||||
PY
|
||||
export MSSQL_CONTAINER MSSQL_USER MSSQL_PASS
|
||||
gp() { python3 "$HELPER" "$@"; }
|
||||
jget() { python3 -c 'import json,sys
|
||||
d=json.loads(sys.argv[1])
|
||||
for k in sys.argv[2].split("."):
|
||||
if d is None: break
|
||||
d = d.get(k) if isinstance(d, dict) else None
|
||||
print("" if d is None else (json.dumps(d) if isinstance(d,(dict,list,bool)) else d))' "$1" "$2"; }
|
||||
|
||||
UNWRAP='
|
||||
import json,sys
|
||||
d=json.load(sys.stdin)
|
||||
if isinstance(d, dict) and "data" in d and len(d) <= 2:
|
||||
d = d["data"]
|
||||
'
|
||||
|
||||
site_id() {
|
||||
"$CLI" site list "${A[@]}" | python3 -c "$UNWRAP"'
|
||||
import sys
|
||||
print(next(str(s["id"]) for s in d if s["siteIdentifier"]==sys.argv[1]))' "$1"
|
||||
}
|
||||
site_hash() {
|
||||
"$CLI" site list "${A[@]}" | python3 -c "$UNWRAP"'
|
||||
import sys
|
||||
print(next((s.get("artifactRevisionHash") or "-") for s in d if s["siteIdentifier"]==sys.argv[1]))' "$1"
|
||||
}
|
||||
conn_id() { # conn_id <name> -> id or empty
|
||||
"$CLI" data-connection list "${A[@]}" | python3 -c "$UNWRAP"'
|
||||
import sys
|
||||
print(next((str(c["id"]) for c in d if c["name"]==sys.argv[1]), ""))' "$1"
|
||||
}
|
||||
mxgw_config() { # mxgw_config <apiKeyValue>
|
||||
python3 -c 'import json,sys; print(json.dumps({"Endpoint":"http://sb-gate-no-such-gateway:5000","ApiKey":sys.argv[1],"ClientName":"sb-gate","ReadTimeoutMs":5000}))' "$1"
|
||||
}
|
||||
upsert_conn() { # upsert_conn <name> <siteId> <apiKeyValue>
|
||||
local id; id=$(conn_id "$1")
|
||||
if [ -z "$id" ]; then
|
||||
"$CLI" data-connection create "${A[@]}" --site-id "$2" --name "$1" --protocol MxGateway \
|
||||
--primary-config "$(mxgw_config "$3")" >/dev/null
|
||||
else
|
||||
"$CLI" data-connection update "${A[@]}" --id "$id" --name "$1" --protocol MxGateway \
|
||||
--primary-config "$(mxgw_config "$3")" >/dev/null
|
||||
fi
|
||||
}
|
||||
# deploy <siteId|all> -> sets DEPLOY_OUT (raw CLI output) and DEPLOY_RC; never aborts the script
|
||||
DEPLOY_OUT=""; DEPLOY_RC=0
|
||||
deploy() {
|
||||
DEPLOY_RC=0
|
||||
if [ "$1" = "all" ]; then
|
||||
DEPLOY_OUT=$("$CLI" site deploy-artifacts "${A[@]}" 2>&1) || DEPLOY_RC=$?
|
||||
else
|
||||
DEPLOY_OUT=$("$CLI" site deploy-artifacts "${A[@]}" --site-id "$1" 2>&1) || DEPLOY_RC=$?
|
||||
fi
|
||||
}
|
||||
# site_result <siteIdentifier> <field> -> the field from DEPLOY_OUT's siteResults
|
||||
site_result() {
|
||||
printf '%s' "$DEPLOY_OUT" | python3 -c '
|
||||
import json,sys
|
||||
raw=sys.stdin.read()
|
||||
try:
|
||||
d=json.loads(raw[raw.index("{"):])
|
||||
except Exception:
|
||||
print(""); sys.exit(0)
|
||||
r=next((x for x in d.get("siteResults",[]) if x.get("siteId")==sys.argv[1]), None)
|
||||
v=None if r is None else r.get(sys.argv[2])
|
||||
print("" if v is None else (json.dumps(v) if isinstance(v,bool) else v))' "$1" "$2"
|
||||
}
|
||||
badge_of() { jget "$1" "$2"; }
|
||||
|
||||
ensure_secrets_cli() {
|
||||
[ -f "$SECRETS_CLI_DIR/ZB.MOM.WW.Secrets.Cli.dll" ] && return 0
|
||||
echo " publishing the ZB.MOM.WW.Secrets 'secret' CLI to $SECRETS_CLI_DIR ..."
|
||||
dotnet publish "$SECRETS_CLI_PROJECT" -c Release -o "$SECRETS_CLI_DIR" >/dev/null
|
||||
}
|
||||
# Runs the library's own `secret` CLI against site-a-a's LOCAL store, in a throwaway container that
|
||||
# shares that node's data volume and carries that node's KEK — the library's normal SQLite write path.
|
||||
site_a_secret_cli() {
|
||||
local verb="$1"; shift
|
||||
docker run --rm --volumes-from "$ACTIVE_A" -v "$SECRETS_CLI_DIR:/cli:ro" -w /cli \
|
||||
-e Secrets__SqlitePath=/data/scadabridge-secrets.db -e "ZB_SECRETS_MASTER_KEY=$SITE_A_KEK" \
|
||||
-e SB_GATE_VALUE --entrypoint sh scadabridge:latest \
|
||||
-c "exec dotnet /cli/ZB.MOM.WW.Secrets.Cli.dll $verb \"\$@\"" _ "$@"
|
||||
}
|
||||
# `secret set <name> <value>` takes the value positionally: hand it over through the container's
|
||||
# environment (SB_GATE_VALUE) so it never appears on this host's argv; the container's inner sh
|
||||
# expands it.
|
||||
site_a_secret_set() { # site_a_secret_set <name> <description> (value in $SB_GATE_VALUE)
|
||||
docker run --rm --volumes-from "$ACTIVE_A" -v "$SECRETS_CLI_DIR:/cli:ro" -w /cli \
|
||||
-e Secrets__SqlitePath=/data/scadabridge-secrets.db -e "ZB_SECRETS_MASTER_KEY=$SITE_A_KEK" \
|
||||
-e SB_GATE_VALUE --entrypoint sh scadabridge:latest \
|
||||
-c 'exec dotnet /cli/ZB.MOM.WW.Secrets.Cli.dll set "$1" "$SB_GATE_VALUE" --description "$2"' _ "$1" "$2"
|
||||
}
|
||||
|
||||
# ── teardown (idempotent; also the pre-clean) ──────────────────────────────────────────────────
|
||||
teardown() {
|
||||
local rc=$?
|
||||
set +e
|
||||
echo ""
|
||||
echo "=== teardown ==="
|
||||
local id
|
||||
for c in "$CONN_A" "$CONN_B"; do
|
||||
id=$(conn_id "$c" 2>/dev/null)
|
||||
if [ -n "$id" ]; then
|
||||
"$CLI" data-connection delete "${A[@]}" --id "$id" >/dev/null 2>&1 \
|
||||
&& echo " removed data connection $c (id=$id)" || echo " WARN could not remove data connection $c"
|
||||
fi
|
||||
done
|
||||
# Removal must reach both sites that ever held the gate secret BEFORE central forgets it.
|
||||
[ -n "${SITE_A_ID:-}" ] && { deploy "$SITE_A_ID"; echo " site-a redeploy rc=$DEPLOY_RC"; }
|
||||
[ -n "${SITE_B_ID:-}" ] && { deploy "$SITE_B_ID"; echo " site-b redeploy rc=$DEPLOY_RC"; }
|
||||
echo " central $SECRET: $(gp secret-delete "$URL" "$SECRET" 2>&1 | tail -1)"
|
||||
if [ -f "$SECRETS_CLI_DIR/ZB.MOM.WW.Secrets.Cli.dll" ]; then
|
||||
local live
|
||||
live=$(gp site-secrets "$ACTIVE_A" 2>/dev/null | python3 -c 'import json,sys; d=json.load(sys.stdin); print("1" if d.get(sys.argv[1],{}).get("deleted") is False else "")' "$OOB" 2>/dev/null)
|
||||
if [ -n "$live" ]; then
|
||||
site_a_secret_cli rm "$OOB" >/dev/null 2>&1 && echo " removed out-of-band $OOB from site-a-a" \
|
||||
|| echo " WARN could not remove out-of-band $OOB"
|
||||
fi
|
||||
fi
|
||||
deploy all; echo " fleet-wide artifact redeploy rc=$DEPLOY_RC"
|
||||
echo " badges after teardown: $(gp badges "$URL" 2>&1 | tail -1)"
|
||||
[ "${1:-}" = "pre" ] || rm -rf "$WORK"
|
||||
return $rc
|
||||
}
|
||||
|
||||
echo "=== ScadaBridge site-secrets artifact-deploy live gate ==="
|
||||
echo "management url: $URL"
|
||||
GATE_START=$(date -u +%Y-%m-%dT%H:%M:%SZ)
|
||||
SITE_A_ID=$(site_id site-a); SITE_B_ID=$(site_id site-b); SITE_C_ID=$(site_id site-c)
|
||||
echo "sites: site-a=$SITE_A_ID site-b=$SITE_B_ID site-c=$SITE_C_ID"
|
||||
ensure_secrets_cli
|
||||
|
||||
echo ""
|
||||
echo "=== pre-clean (idempotent rerun) ==="
|
||||
teardown pre || true
|
||||
trap teardown EXIT
|
||||
GATE_START=$(date -u +%Y-%m-%dT%H:%M:%SZ)
|
||||
# Docker log checkpoint for check 8: only lines written during this run are grepped.
|
||||
LOG_SINCE="$GATE_START"
|
||||
|
||||
BASE=$(gp badges "$URL")
|
||||
echo "baseline badges: $BASE"
|
||||
for s in site-a site-b site-c; do
|
||||
[ "$(badge_of "$BASE" "$s")" = "current" ] || { echo "baseline: $s is not Current — the gate needs a Current rig to start"; exit 1; }
|
||||
done
|
||||
|
||||
# ── 1 ────────────────────────────────────────────────────────────────────────────────────────
|
||||
begin_check 1 "secret + referencing connection on site-a -> site-a Out of date, site-b/c unchanged"
|
||||
R=$(SB_GATE_VALUE="$VALUE_1" gp secret-upsert "$URL" "$SECRET" "site-secrets live gate (throwaway)")
|
||||
[ "$R" = "add" ] && ok "created $SECRET at central via /admin/secrets" || bad "create $SECRET: $R"
|
||||
C1=$(gp central-secret "$SECRET")
|
||||
[ "$(jget "$C1" deleted)" = "false" ] && ok "central row $SECRET r$(jget "$C1" rev) kek $(jget "$C1" kek) sha $(jget "$C1" sha | cut -c1-12)" \
|
||||
|| bad "central row for $SECRET not found live"
|
||||
H_A_BEFORE=$(site_hash site-a)
|
||||
upsert_conn "$CONN_A" "$SITE_A_ID" "secret:$SECRET" && ok "created MxGateway connection $CONN_A on site-a (ApiKey=secret:$SECRET, unbound)" \
|
||||
|| bad "create $CONN_A"
|
||||
B=$(gp badges "$URL"); echo " badges: $B"
|
||||
[ "$(badge_of "$B" site-a)" = "out of date" ] && ok "site-a Out of date" || bad "site-a is '$(badge_of "$B" site-a)', expected out of date"
|
||||
[ "$(badge_of "$B" site-b)" = "current" ] && ok "site-b Current (unchanged)" || bad "site-b is '$(badge_of "$B" site-b)'"
|
||||
[ "$(badge_of "$B" site-c)" = "current" ] && ok "site-c Current (unchanged)" || bad "site-c is '$(badge_of "$B" site-c)'"
|
||||
end_check 1
|
||||
|
||||
# ── 2 ────────────────────────────────────────────────────────────────────────────────────────
|
||||
begin_check 2 "deploy artifacts to site-a -> success, secretsDeployed 1, site-a Current"
|
||||
T0=$(python3 -c 'import time; print(time.time())')
|
||||
deploy "$SITE_A_ID"
|
||||
T_DEPLOYED=$(python3 -c 'import time; print(time.time())')
|
||||
echo " deploy rc=$DEPLOY_RC ($(python3 -c "print(round($T_DEPLOYED-$T0,1))") s): $(printf '%s' "$DEPLOY_OUT" | tr -d '\n' | cut -c1-240)"
|
||||
[ "$DEPLOY_RC" -eq 0 ] && [ "$(site_result site-a success)" = "true" ] && ok "site-a deploy succeeded" || bad "site-a deploy failed (rc=$DEPLOY_RC)"
|
||||
[ "$(site_result site-a secretsDeployed)" = "1" ] && ok "secretsDeployed: 1" || bad "secretsDeployed='$(site_result site-a secretsDeployed)', expected 1"
|
||||
B=$(gp badges "$URL"); echo " badges: $B"
|
||||
[ "$(badge_of "$B" site-a)" = "current" ] && ok "site-a Current" || bad "site-a is '$(badge_of "$B" site-a)', expected current"
|
||||
end_check 2
|
||||
|
||||
# wait_node <container> <python predicate over the site-secrets JSON d> -> sets WAIT_S (or empty on timeout)
|
||||
WAIT_S=""
|
||||
wait_node() {
|
||||
local c="$1" pred="$2" start now j
|
||||
start=$(python3 -c 'import time; print(time.time())'); WAIT_S=""
|
||||
while :; do
|
||||
j=$(gp site-secrets "$c")
|
||||
if python3 -c "import json,sys; d=json.loads(sys.argv[1]); sys.exit(0 if ($pred) else 1)" "$j"; then
|
||||
now=$(python3 -c 'import time; print(time.time())')
|
||||
WAIT_S=$(python3 -c "print(round($now-${3:-$start},1))"); LAST_NODE_JSON="$j"; return 0
|
||||
fi
|
||||
now=$(python3 -c 'import time; print(time.time())')
|
||||
python3 -c "import sys; sys.exit(0 if $now-$start < $STANDBY_WINDOW_S else 1)" || { LAST_NODE_JSON="$j"; return 1; }
|
||||
sleep 1
|
||||
done
|
||||
}
|
||||
matches_central() { # python predicate: node row == central row C (rev + sha), live
|
||||
echo "d.get('$SECRET') is not None and not d['$SECRET']['deleted'] and d['$SECRET']['rev']==$(jget "$1" rev) and d['$SECRET']['sha']=='$(jget "$1" sha)' and d['$SECRET']['kek']=='$(jget "$1" kek)'"
|
||||
}
|
||||
|
||||
# ── 3 ────────────────────────────────────────────────────────────────────────────────────────
|
||||
begin_check 3 "both site-a nodes hold $SECRET at the central revision (standby within ${STANDBY_WINDOW_S}s)"
|
||||
C=$(gp central-secret "$SECRET")
|
||||
PRED=$(matches_central "$C")
|
||||
if wait_node "$ACTIVE_A" "$PRED" "$T_DEPLOYED"; then ok "site-a-a (active) r$(jget "$C" rev), ciphertext sha256 $(jget "$C" sha | cut -c1-16)… == central (seen ${WAIT_S}s after the deploy returned)"
|
||||
else bad "site-a-a does not match central: $LAST_NODE_JSON"; fi
|
||||
if wait_node "$STANDBY_A" "$PRED" "$T_DEPLOYED"; then STANDBY_3="$WAIT_S"; ok "site-a-b (standby) converged ${WAIT_S}s after the deploy returned"
|
||||
else STANDBY_3="timeout"; bad "site-a-b did not converge within ${STANDBY_WINDOW_S}s: $LAST_NODE_JSON"; fi
|
||||
L=$(gp site-localdb "$STANDBY_A")
|
||||
[ "$(jget "$L" "mirrored.$SECRET")" = "$(jget "$C" rev)" ] && ok "site-a-b mirrored_secrets ledger r$(jget "$C" rev)" || bad "site-a-b ledger: $L"
|
||||
end_check 3
|
||||
|
||||
# ── 4 ────────────────────────────────────────────────────────────────────────────────────────
|
||||
begin_check 4 "rotate at central -> site-a Out of date -> deploy -> both nodes at the new revision"
|
||||
R=$(SB_GATE_VALUE="$VALUE_2" gp secret-upsert "$URL" "$SECRET" "site-secrets live gate (throwaway, rotated)")
|
||||
[ "$R" = "rotate" ] && ok "rotated $SECRET via /admin/secrets" || bad "rotate: $R"
|
||||
C2=$(gp central-secret "$SECRET")
|
||||
if [ "$(jget "$C2" rev)" -gt "$(jget "$C1" rev)" ] && [ "$(jget "$C2" sha)" != "$(jget "$C1" sha)" ]; then
|
||||
ok "central r$(jget "$C1" rev) -> r$(jget "$C2" rev), ciphertext changed"
|
||||
else bad "central row did not advance: before r$(jget "$C1" rev) after r$(jget "$C2" rev)"; fi
|
||||
B=$(gp badges "$URL"); echo " badges: $B"
|
||||
[ "$(badge_of "$B" site-a)" = "out of date" ] && ok "site-a Out of date after the rotation" || bad "site-a is '$(badge_of "$B" site-a)'"
|
||||
T0=$(python3 -c 'import time; print(time.time())'); deploy "$SITE_A_ID"; T_DEPLOYED=$(python3 -c 'import time; print(time.time())')
|
||||
[ "$DEPLOY_RC" -eq 0 ] && [ "$(site_result site-a success)" = "true" ] && [ "$(site_result site-a secretsDeployed)" = "1" ] \
|
||||
&& ok "redeploy succeeded, secretsDeployed 1" || bad "redeploy: rc=$DEPLOY_RC $(printf '%s' "$DEPLOY_OUT" | tr -d '\n' | cut -c1-240)"
|
||||
PRED=$(matches_central "$C2")
|
||||
if wait_node "$ACTIVE_A" "$PRED" "$T_DEPLOYED"; then ok "site-a-a at r$(jget "$C2" rev) (${WAIT_S}s)"; else bad "site-a-a: $LAST_NODE_JSON"; fi
|
||||
if wait_node "$STANDBY_A" "$PRED" "$T_DEPLOYED"; then STANDBY_4="$WAIT_S"; ok "site-a-b at r$(jget "$C2" rev) (${WAIT_S}s after the deploy returned)"
|
||||
else STANDBY_4="timeout"; bad "site-a-b: $LAST_NODE_JSON"; fi
|
||||
B=$(gp badges "$URL")
|
||||
[ "$(badge_of "$B" site-a)" = "current" ] && ok "site-a Current again" || bad "site-a is '$(badge_of "$B" site-a)'"
|
||||
end_check 4
|
||||
|
||||
# ── 5 ────────────────────────────────────────────────────────────────────────────────────────
|
||||
begin_check 5 "drop the secret: reference -> deploy -> both nodes drop $SECRET; out-of-band $OOB survives"
|
||||
SB_GATE_VALUE="$VALUE_OOB" site_a_secret_set "$OOB" "site-secrets gate out-of-band" >/dev/null 2>&1 || true
|
||||
J=$(gp site-secrets "$ACTIVE_A")
|
||||
if [ "$(python3 -c 'import json,sys; d=json.loads(sys.argv[1]); print(d.get(sys.argv[2],{}).get("deleted"))' "$J" "$OOB")" = "False" ]; then
|
||||
ok "seeded out-of-band $OOB on site-a-a through the library's own write path"
|
||||
else bad "out-of-band seed not visible on site-a-a: $J"; fi
|
||||
upsert_conn "$CONN_A" "$SITE_A_ID" "gate-literal-not-a-secret-ref" && ok "$CONN_A no longer references a secret" || bad "update $CONN_A"
|
||||
T0=$(python3 -c 'import time; print(time.time())'); deploy "$SITE_A_ID"; T_DEPLOYED=$(python3 -c 'import time; print(time.time())')
|
||||
[ "$DEPLOY_RC" -eq 0 ] && [ "$(site_result site-a success)" = "true" ] && ok "deploy succeeded (secretsDeployed='$(site_result site-a secretsDeployed)')" \
|
||||
|| bad "deploy: rc=$DEPLOY_RC $(printf '%s' "$DEPLOY_OUT" | tr -d '\n' | cut -c1-240)"
|
||||
GONE="(d.get('$SECRET') is None or d['$SECRET']['deleted'])"
|
||||
if wait_node "$ACTIVE_A" "$GONE" "$T_DEPLOYED"; then ok "site-a-a no longer holds $SECRET (${WAIT_S}s)"; else bad "site-a-a still holds $SECRET: $LAST_NODE_JSON"; fi
|
||||
if wait_node "$STANDBY_A" "$GONE" "$T_DEPLOYED"; then STANDBY_5="$WAIT_S"; ok "site-a-b no longer holds $SECRET (${WAIT_S}s after the deploy returned)"
|
||||
else STANDBY_5="timeout"; bad "site-a-b still holds $SECRET: $LAST_NODE_JSON"; fi
|
||||
J=$(gp site-secrets "$ACTIVE_A")
|
||||
[ "$(python3 -c 'import json,sys; d=json.loads(sys.argv[1]); print(d.get(sys.argv[2],{}).get("deleted"))' "$J" "$OOB")" = "False" ] \
|
||||
&& ok "out-of-band $OOB still live on site-a-a" || bad "out-of-band $OOB lost: $J"
|
||||
L=$(gp site-localdb "$ACTIVE_A")
|
||||
[ "$(jget "$L" "deployed.$SECRET.removed")" = "true" ] && ok "deployed_secrets.$SECRET is_removed=1 (kept for the standby)" || bad "deployed_secrets: $L"
|
||||
end_check 5
|
||||
|
||||
# ── 6 ────────────────────────────────────────────────────────────────────────────────────────
|
||||
begin_check 6 "reference non-existent $MISSING -> site-a deploy fails closed, stays Out of date, site untouched"
|
||||
SNAP_BEFORE="$(site_hash site-a)|$(gp site-secrets "$ACTIVE_A")|$(gp site-localdb "$ACTIVE_A")|$(gp site-secrets "$STANDBY_A")|$(gp site-localdb "$STANDBY_A")"
|
||||
upsert_conn "$CONN_A" "$SITE_A_ID" "secret:$MISSING" && ok "$CONN_A now references secret:$MISSING" || bad "update $CONN_A"
|
||||
deploy "$SITE_A_ID"
|
||||
ERR=$(site_result site-a errorMessage)
|
||||
echo " deploy rc=$DEPLOY_RC error: $ERR"
|
||||
EXPECT="Secret '$MISSING' referenced by data connection(s) $CONN_A is not in the central secret store"
|
||||
[ "$(site_result site-a success)" = "false" ] && ok "site-a deploy reported failure" || bad "site-a deploy did not fail: $(printf '%s' "$DEPLOY_OUT" | tr -d '\n' | cut -c1-240)"
|
||||
case "$ERR" in *"$EXPECT"*) ok "named fail-closed message";; *) bad "message does not contain: $EXPECT";; esac
|
||||
B=$(gp badges "$URL"); echo " badges: $B"
|
||||
[ "$(badge_of "$B" site-a)" = "out of date" ] && ok "site-a stays Out of date" || bad "site-a is '$(badge_of "$B" site-a)'"
|
||||
sleep 3
|
||||
SNAP_AFTER="$(site_hash site-a)|$(gp site-secrets "$ACTIVE_A")|$(gp site-localdb "$ACTIVE_A")|$(gp site-secrets "$STANDBY_A")|$(gp site-localdb "$STANDBY_A")"
|
||||
[ "$SNAP_BEFORE" = "$SNAP_AFTER" ] && ok "nothing changed: stamped hash, both nodes' secret rows, deployed_secrets, ledgers and the $CONN_A definition identical" \
|
||||
|| { bad "site state changed"; echo " before: $SNAP_BEFORE"; echo " after: $SNAP_AFTER"; }
|
||||
end_check 6
|
||||
|
||||
# ── 7 ────────────────────────────────────────────────────────────────────────────────────────
|
||||
begin_check 7 "the same reference on site-b (mismatched dev KEK) -> deploy fails naming the KekId"
|
||||
C=$(gp central-secret "$SECRET")
|
||||
KEK=$(jget "$C" kek)
|
||||
upsert_conn "$CONN_B" "$SITE_B_ID" "secret:$SECRET" && ok "created $CONN_B on site-b (ApiKey=secret:$SECRET)" || bad "create $CONN_B"
|
||||
deploy "$SITE_B_ID"
|
||||
ERR=$(site_result site-b errorMessage)
|
||||
echo " deploy rc=$DEPLOY_RC error: $ERR"
|
||||
[ "$(site_result site-b success)" = "false" ] && ok "site-b deploy reported failure" || bad "site-b deploy did not fail: $(printf '%s' "$DEPLOY_OUT" | tr -d '\n' | cut -c1-240)"
|
||||
case "$ERR" in *"$SECRET"*"$KEK"*) ok "message names $SECRET and the row's KekId $KEK";; *) bad "message does not name $SECRET + KekId $KEK";; esac
|
||||
B=$(gp badges "$URL"); echo " badges: $B"
|
||||
[ "$(badge_of "$B" site-b)" = "out of date" ] && ok "site-b not stamped (Out of date)" || bad "site-b is '$(badge_of "$B" site-b)'"
|
||||
end_check 7
|
||||
|
||||
# ── 8 ────────────────────────────────────────────────────────────────────────────────────────
|
||||
begin_check 8 "DeploySiteSecrets audit rows; no plaintext / base64 ciphertext in any node log"
|
||||
AUD=$("$CLI" audit-config query "${A[@]}" --action DeploySiteSecrets --entity-type SiteSecrets --from "$GATE_START" --page-size 50 2>/dev/null)
|
||||
ROWS=$(printf '%s' "$AUD" | python3 -c '
|
||||
import json,sys
|
||||
d=json.load(sys.stdin)
|
||||
rows=[]
|
||||
for e in sorted(d.get("entries",[]), key=lambda e: e["timestamp"]):
|
||||
s=json.loads(e.get("afterStateJson") or "{}")
|
||||
names=",".join("%s@r%s" % (x.get("name"), x.get("revision")) for x in (s.get("secrets") or []))
|
||||
miss=",".join(s.get("missing") or [])
|
||||
rows.append("%s|%s|%s|%s|%s" % (e["entityId"], s.get("success"), names or "-", miss or "-", s.get("deploymentId")))
|
||||
print("\n".join(rows))')
|
||||
echo "$ROWS" | sed 's/^/ audit: /'
|
||||
R2=$(jget "$C1" rev); R4=$(jget "$C2" rev)
|
||||
EXPECTED="site-a|True|$SECRET@r$R2|-
|
||||
site-a|True|$SECRET@r$R4|-
|
||||
site-a|False|-|$MISSING
|
||||
site-b|False|$SECRET@r$R4|-"
|
||||
GOT=$(printf '%s\n' "$ROWS" | cut -d'|' -f1-4)
|
||||
[ "$GOT" = "$EXPECTED" ] && ok "exactly 4 rows, one per secret-bearing deploy (2, 4, 6-fail-closed, 7-kek-mismatch); the no-secret deploy of check 5 wrote none" \
|
||||
|| { bad "audit rows differ from expected"; echo " expected:"; echo "$EXPECTED" | sed 's/^/ /'; }
|
||||
[ "$(printf '%s\n' "$ROWS" | cut -d'|' -f5 | sort -u | grep -c .)" = "4" ] && ok "4 distinct deploymentIds" || bad "deploymentIds not distinct"
|
||||
# Material to hunt for: both plaintext values and both ciphertexts (base64, as they appear in any
|
||||
# JSON serialization). Written to a pattern file so they never reach argv or the output.
|
||||
PAT="$WORK/patterns"; : >"$PAT"; chmod 600 "$PAT"
|
||||
printf '%s\n%s\n%s\n' "$VALUE_1" "$VALUE_2" "$VALUE_OOB" >>"$PAT"
|
||||
jget "$C1" b64 >>"$PAT"; jget "$C2" b64 >>"$PAT"
|
||||
[ "$(grep -c . "$PAT")" = "5" ] && [ "$(awk 'length($0) < 20' "$PAT" | grep -c .)" = "0" ] \
|
||||
&& ok "5 material patterns (3 plaintext, 2 base64 ciphertext) armed" || bad "pattern file malformed"
|
||||
HITS=0
|
||||
for n in "${NODES[@]}"; do
|
||||
h=$(docker logs --since "$LOG_SINCE" "scadabridge-$n" 2>&1 | grep -c -F -f "$PAT" || true)
|
||||
HITS=$((HITS + h)); [ "$h" = "0" ] || echo " scadabridge-$n: $h hit(s)"
|
||||
done
|
||||
AUDH=$(printf '%s' "$AUD" | grep -c -F -f "$PAT" || true)
|
||||
[ "$HITS" = "0" ] && ok "0 hits across all 8 nodes' logs since $LOG_SINCE" || bad "$HITS log line(s) carry secret material"
|
||||
[ "$AUDH" = "0" ] && ok "0 hits in the DeploySiteSecrets audit payloads" || bad "audit payload carries secret material"
|
||||
end_check 8
|
||||
|
||||
echo ""
|
||||
echo "=== Summary ==="
|
||||
echo "standby convergence (site-a-b, seconds after the deploy call returned): check3=${STANDBY_3:-?} check4=${STANDBY_4:-?} check5=${STANDBY_5:-?}"
|
||||
echo "PASS $CHECKS_PASSED/8"
|
||||
[ "$CHECKS_FAILED" -eq 0 ] || exit 1
|
||||
@@ -120,6 +120,14 @@ Both central nodes share state through:
|
||||
because a failover then 401s every inbound client until keys are re-issued on the survivor. Both
|
||||
docker rigs run `SqlServer`; see `docs/operations/inbound-api-key-reissue.md` → *Central pairs* to
|
||||
migrate an existing pair.
|
||||
- **Secrets store**: shared **only** when `Secrets:Store:Provider = SqlServer` (design 2026-09-28,
|
||||
`docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`) — both nodes then read/write the
|
||||
same shared SQL Server secret store, so `/admin/secrets` create/rotate/delete changes are
|
||||
cluster-wide immediately and both nodes deploy the same secret set to sites regardless of which
|
||||
is active. The library default, `Provider = Sqlite`, stores secrets in a **per-node** file instead;
|
||||
running `Sqlite` on a central pair is a documented misconfiguration — `StartupValidator` logs a
|
||||
boot-time Warning. See *Secrets: shared store, per-site deploy, and the KEK* below for the KEK
|
||||
requirement this store does **not** by itself satisfy.
|
||||
|
||||
### Load Balancer
|
||||
|
||||
@@ -558,6 +566,59 @@ path. That is the accepted posture today — the same trusted-network assumption
|
||||
already made, now with authentication rather than none. TLS on these listeners is follow-on
|
||||
hardening and needs no change to the key design.
|
||||
|
||||
#### Secrets: shared store, per-site deploy, and the KEK
|
||||
|
||||
Site secrets ride the artifact deploy (2026-09-28, `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`;
|
||||
see that doc's *Implementation notes (as built)* for where the shipped code deviates from the
|
||||
original design). This section is the operational summary; component detail lives in
|
||||
Component-Host.md → *Secrets*, Component-DeploymentManager.md → *Site secrets*, and
|
||||
Component-SiteRuntime.md → *Deployed Secrets*.
|
||||
|
||||
- **Central**: `Secrets:Store:Provider = SqlServer` on both central nodes — see *Shared State*
|
||||
above. `SiteSecretsCollector` resolves each site's referenced secrets from that store at deploy
|
||||
time; a missing/tombstoned secret fails that site's deploy before anything is sent.
|
||||
- **Every node must share the same KEK** (`ZB_SECRETS_MASTER_KEY` / master-key file) — **every
|
||||
central node and every site node**, not just the central pair. This is the same requirement the
|
||||
retired gRPC secrets hub had; it did not change. **Nothing checks this at boot or at the next
|
||||
reconnect** — a mismatch surfaces only at **deploy time**, on the site: `DeployedSecretsMirror`'s
|
||||
verify step trial-decrypts every applied secret and reports a distinct error naming the secret
|
||||
and its `KekId` if the local master key cannot open it. Verify the KEK is identical everywhere
|
||||
**before** turning this on for a site — the rig's `docker/docker-compose.yml` deliberately runs
|
||||
site-b/site-c on a mismatched KEK precisely to exercise this failure path live.
|
||||
- **Standby / passive-node convergence.** The active site node applies a deploy's secrets
|
||||
synchronously (so the deploy response reflects that apply immediately); the standby node never
|
||||
receives that synchronous call — it converges through ordinary LocalDb replication of the
|
||||
`deployed_secrets` table plus its **own** `DeployedSecretsMirror` running on its own schedule,
|
||||
`ScadaBridge:SiteRuntime:DeployedSecretsMirrorInterval` (default **15s**). A freshly-promoted
|
||||
standby is therefore current on secrets within one interval of the last deploy, not instantly.
|
||||
- **A site pair that uses deployed secrets MUST run LocalDb replication** (`LocalDb:Replication:*`
|
||||
configured on both nodes — see *Reading the replication backlog*). The standby's only path to the
|
||||
secrets is the replicated `deployed_secrets` table. On an **unreplicated** pair only the node that
|
||||
was active at deploy time holds the secrets; after a failover the new active node has **none**
|
||||
(or a stale hand-seeded copy) until the next *Deploy Artifacts*, and a removal applied while one
|
||||
node was active leaves the secret live on the other. The Artifacts badge still reads *Current* —
|
||||
the stamp is per site, not per node. If a pair cannot be replicated, re-run *Deploy Artifacts*
|
||||
after every failover.
|
||||
- **Secret rotation and live connections.** An artifact deploy writes and mirrors the shipped
|
||||
secrets **before** it pushes data-connection definitions to the DCL, so a connection created or
|
||||
reconfigured by the same deploy resolves a newly shipped secret on its first connect. A rotation
|
||||
that leaves the connection's config unchanged does **not** recreate the connection (the push is
|
||||
hash-guarded): the new value takes effect on that connection's **next (re)connect**.
|
||||
- **Rollout is a deployment record, not a code change** — see the deferred-work register (row 127)
|
||||
for the vd03/QA prerequisites (shared store provisioning, `Provider=SqlServer` on both centrals,
|
||||
the KEK alignment check above, LocalDb replication on every site pair that uses secrets,
|
||||
recreating any hand-seeded site secrets centrally, then *Deploy Artifacts*).
|
||||
- **Upgrade impact — do the prerequisites in the same window as the upgrade.** Once a central runs
|
||||
this build, any site whose data connections carry a `secret:` reference fails its **whole**
|
||||
artifact deploy (shared scripts, external systems and DB connections included, not just secrets)
|
||||
and its Artifacts badge reads *Out of date*, until every referenced secret exists in the central
|
||||
store. Before upgrading an environment, check whether its connection configs use `secret:`
|
||||
references and whether its site pairs run LocalDb replication.
|
||||
- **The gRPC secrets hub is retired in ScadaBridge** — `Secrets:Replication:Enabled` now fails
|
||||
startup on either role (Component-Host.md → *Secrets*). QA and vd03 currently run
|
||||
`Replication:Enabled=false`, so the boot itself will not fail fast — the deploy-time impact above
|
||||
is what bites.
|
||||
|
||||
## Scaling Guidelines
|
||||
|
||||
### Target Scale
|
||||
|
||||
@@ -120,6 +120,7 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round
|
||||
| 124 | **No template-graph-watermark short-circuit on the pending-updates fleet flatten** *(Low)*. `PendingUpdatesEvaluator.EvaluateAsync` re-flattens every deployed instance through `DeploymentService.GetInstanceStalenessAsync` on every cache-TTL expiry (`PendingUpdatesCacheTtl`, default 60 s) while any Deployer/Admin circuit is open, even when nothing in the template/instance/artifact graph has changed since the last evaluation. | `docs/plans/2026-09-14-pending-updates-design.md` §3.2 ("Not in scope") | The single-flight TTL cache already bounds the cost to one fleet flatten per TTL fleet-wide (not per viewer); a watermark short-circuit is a further optimization, not a correctness need, and needs a cheap "has anything changed" signal that does not exist yet. | The fleet flatten's cost (instance count x template depth) becomes measurable against `PendingUpdatesCacheTtl` on a large deployment. |
|
||||
| 125 | **Pending-updates counts have no Health-dashboard tile or CLI surface** *(Low)*. The aggregate (`PendingUpdatesSnapshot.StaleInstanceCount` / `StaleArtifactSiteCount`) is consumed only by the Central UI Sites page and rail indicator; there is no Health dashboard KPI tile and no `deploy`/`site` CLI command surfacing it — `site list` shows only the raw `artifactRevisionHash`/`artifactsDeployedAt` fields on each site, not whether they are current. | `docs/plans/2026-09-14-pending-updates-design.md` §3.2 ("Not in scope") | Out of scope for the UI-focused ask; the CLI/Health surfaces are a separate consumer of the same `IPendingUpdatesCache`. | An operator wants the pending-updates counts without opening the Central UI (headless monitoring, CLI scripting). |
|
||||
| 126 | ~~**TODO — roll the two outstanding migrations to the QA pair vq03/vq04**~~ **DONE 2026-09-28** — QA vq03/vq04 upgraded `2b9a7c3c` → `b3021e7c`, 55 → 57, artifacts deployed (record `docs/deployment/deployment-records/2026-09-28-wonder-app-vq03-vq04-b3021e7c.md`). *(deployment, closed)*. **DEV vd03 DONE 2026-09-28** (`fe89363b`, 55 → 57, record `docs/deployment/deployment-records/2026-09-28-wonder-app-vd03-fe89363b.md` — the `sb-mig-55to57.sql` there is the same script QA needs). QA still runs `2b9a7c3c` with the DB at migration 55 (`20260909203046_EnforceGlobalInstanceNamesAndProtectProtocolConfiguration`); `main` (`ac833f5a`, pushed 2026-09-14) carries two more: **56** `20260911135501_AddAlarmMesReportable` (computed-alarm `IsMesReportable` flag) and **57** `20260914173147_AddSiteArtifactRevision` (`Sites.ArtifactRevisionHash`/`ArtifactsDeployedAt`, both nullable). Both are additive `AddColumn`s — no data upgrade, no coordinated step like the 2026-09-10 protocol-config encryption. Recipe (same as the 55 rollout): generate ONE from→to delta script `dotnet ef migrations script EnforceGlobalInstanceNamesAndProtectProtocolConfiguration AddSiteArtifactRevision --project src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase --startup-project src/ZB.MOM.WW.ScadaBridge.Host -o sb-mig-55to57.sql` (NOT idempotent, identical for vd03 and the QA pair), take the pre-upgrade DB backup, stop the pair TOGETHER, apply via `sqlcmd -C -b -I` on the SQL host, publish/swap the `ac833f5a` binaries, start the pair, confirm the Central boot log shows 0 pending migrations and no "Applying migration" (the script already applied them). After upgrade every site reads *Artifacts never deployed* until one *Deploy Artifacts to All Sites* — run it as the last step. Write the deployment record `docs/deployment/deployment-records/<date>-wonder-app-{vd03,vq03-vq04}-ac833f5a.md`. | Pending-updates finish (2026-09-14) | ~~Blocked by the 2026-09-11 remote-access hold~~ — **unblocked 2026-09-28**: `winrm` is re-enabled for deployment activities (CLAUDE.md). vd03 done; QA done 2026-09-28. | Closed 2026-09-28. |
|
||||
| 127 | **TODO — roll out site secrets via artifact deploy to vd03 + QA (vq03/vq04)**. The gRPC secrets hub is retired in ScadaBridge as of `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md` (implemented on branch `feat/site-secrets-artifact-deploy`, rig-only so far); vd03 and QA still run the pre-feature build with `Secrets:Replication:Enabled=false`, so upgrading them will **not** fail the boot (the retirement's `StartupValidator` rejection only fires on `Enabled=true`). **Upgrade impact:** once upgraded, any site whose data connections carry `secret:` references fails its **whole** artifact deploy (scripts, external systems, DB connections too) and reads *Out of date* until every referenced secret exists in the central store — so before upgrading vd03/QA, check whether their connection configs use `secret:` references and whether their site pairs run LocalDb replication, and do the prerequisites in the same window. Prerequisites, in order: (1) provision the `zbsecrets` schema/store in the environment's configuration SQL Server (the same server `ConfigurationDb` already points at); (2) set `Secrets:Store:Provider=SqlServer` on **both** central nodes of the environment; (3) verify the KEK (`ZB_SECRETS_MASTER_KEY` / master-key file) is identical across **every** node — both centrals and every site node — since a mismatch is caught only at deploy time by `DeployedSecretsMirror`'s per-secret verify, not at boot; (4) confirm every site pair that uses secrets runs LocalDb replication — on an unreplicated pair only the node active at deploy time gets the secrets, so a failover leaves the new active node without them until the next *Deploy Artifacts*; (5) recreate, at the new central shared store, any secret an operator previously hand-seeded directly into a site's local `secrets.db` (those never round-tripped through central and would otherwise be silently absent from the referenced-secrets scan). Then run *Deploy Artifacts* against every affected site as the rollout's last step. | `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md` §4 *Rollout*; `docs/deployment/topology-guide.md` → *Secrets: shared store, per-site deploy, and the KEK* | Deployment record, not a code change — vd03/QA config is user-held and their rollout schedule is independent of the branch merging to main. | User asks for the vd03/QA secrets window. |
|
||||
|
||||
## Resolved (verified against the code 2026-07-10)
|
||||
Rows removed from the Deferred table above once confirmed shipped. Kept here for traceability.
|
||||
|
||||
@@ -0,0 +1,249 @@
|
||||
# Site secrets ride the artifact deploy — design
|
||||
|
||||
**Status:** design, 2026-09-28. Plan: `2026-09-28-site-secrets-artifact-deploy.md`.
|
||||
|
||||
## 1. Problem
|
||||
|
||||
A site resolves secrets at runtime (today: an MxGateway data connection's `ApiKey: "secret:<name>"`,
|
||||
`DataConnectionLayer/Adapters/MxGatewayDataConnection.cs:101`). The only way a secret reaches a site
|
||||
is the pull-only gRPC secrets hub (`Secrets:Replication:Mode=Grpc`), which:
|
||||
|
||||
- copies **every** central secret to **every** participating site node every 30 s — no per-site scope;
|
||||
- is invisible to the deployment model: the Sites page *Artifacts current / out of date* badge
|
||||
(`Site.ArtifactRevisionHash`, 2026-09-14) knows nothing about secrets, so a site can read *Current*
|
||||
while its data connection cannot resolve its key;
|
||||
- logs nothing on a successful in-sync sweep (library `GrpcSecretSyncService` logs only when rows were
|
||||
pulled or on failure) and exposes no status;
|
||||
- is enabled only on the docker rig's site-a; vd03 and QA run with replication OFF, so each site node's
|
||||
`secrets.db` is filled by hand.
|
||||
|
||||
## 2. Decisions (user, 2026-09-28)
|
||||
|
||||
| # | Decision |
|
||||
|---|---|
|
||||
| U1 | A site receives **only the secrets it references** — never other sites' keys. |
|
||||
| U2 | The artifact deploy **replaces** the gRPC hub as the way secrets reach sites (hub retired for ScadaBridge sites). |
|
||||
| U3 | Logging = **deploy log lines** (central + site), **audit rows**, and **hub sweep logging** (for the transition until the hub is retired, and for other hub consumers). |
|
||||
| U4 | Central deploys from **one shared SQL-Server secret store** used by both central nodes. |
|
||||
|
||||
## 3. Design
|
||||
|
||||
### 3.1 Which secrets a site needs — derived, not configured
|
||||
|
||||
`SecretReferenceScanner.Scan(IEnumerable<DataConnectionArtifact>)` walks each connection's
|
||||
`PrimaryConfigurationJson` and `BackupConfigurationJson` and collects every JSON **string value**
|
||||
of the form `secret:<name>` (prefix case-insensitive, `<name>` trimmed, must be a valid `SecretName`).
|
||||
It is protocol-agnostic on purpose: a new adapter that adopts `secret:` references is covered with
|
||||
no scanner change. Result: a distinct, ordinal-sorted name set per site, each name carrying the
|
||||
data connections that reference it (for error messages and logs).
|
||||
|
||||
**Deliberately NOT included:** bootstrap secrets — the site's gRPC PSK (`SB-GRPC-PSK-{siteId}`), the
|
||||
LocalDb replication key, the secrets master key. The site needs them *before* it can receive an
|
||||
artifact deploy, and pushing a rotated PSK through the channel it authenticates would sever that
|
||||
channel. They stay out-of-band (service environment / hand-seeded store), exactly as today on vd03/QA.
|
||||
`${secret:X}` references inside a node's own `appsettings` are node-local and invisible to central —
|
||||
also out of scope.
|
||||
|
||||
### 3.2 Central source of truth — the shared SQL store (U4)
|
||||
|
||||
New option `Secrets:Store:Provider` (`Sqlite` default | `SqlServer`), read **only on the Central
|
||||
role**. `SqlServer` registers the library's `AddZbSecretsSqlServerStore` against
|
||||
`Secrets:SqlServer:ConnectionString` (literal/env value — never `${secret:}`, same bootstrap rule as
|
||||
today) so both central nodes read and write one store; the `/admin/secrets` page edits it. This is
|
||||
the store the hub's Grpc-mode central already uses — decoupled from the hub. A clustered Central left
|
||||
on `Sqlite` boots but `StartupValidator` logs a Warning (same shape as the inbound API key-store
|
||||
warning, row 108), because the two nodes would deploy different secret sets depending on which is
|
||||
active.
|
||||
|
||||
### 3.3 The wire: ciphertext rows, never plaintext
|
||||
|
||||
`DeployArtifactsCommand` gains an additive `Secrets` list of `SecretArtifact` — the library's
|
||||
`StoredSecret` fields verbatim (name, content type, ciphertext/nonce/tag, wrapped DEK/nonce/tag,
|
||||
`KekId`, `Revision`, created/updated UTC). Central never decrypts; the site applies the row with
|
||||
`ISecretStore.ApplyReplicatedAsync` (last-writer-wins on `UpdatedUtc`/`Revision`, the same path the
|
||||
hub sweep uses). **Every node must therefore share the KEK** (`ZB_SECRETS_MASTER_KEY` / master-key
|
||||
file) — exactly the hub's existing requirement. A mismatch is detected at deploy time (§3.5), not at
|
||||
the next reconnect. Proto: `DeployArtifactsCommandDto.secrets = 9` (`SecretArtifactListDto`, absent ⇒
|
||||
null) and `ArtifactDeploymentResponseDto` gains `secrets_applied` (wrapper `Int32Value`, absent ⇒ null).
|
||||
|
||||
**Fail-closed on a missing secret.** If a referenced name is absent or tombstoned in the central
|
||||
store, that site's deploy fails before anything is sent: `Secret 'X' referenced by data connection(s)
|
||||
A, B is not in the central secret store`. Shipping the rest would stamp the site *Current* while a
|
||||
connection cannot resolve its key.
|
||||
|
||||
### 3.4 Staleness — secrets are part of the artifact hash
|
||||
|
||||
`ArtifactRevisionHashService` adds `Secrets` → `HashableSecret { ContentDigest, KekId, Missing,
|
||||
Name, Revision }` (alphabetical — the existing ordering test covers it). `ContentDigest` = SHA-256 of
|
||||
ciphertext‖nonce‖tag‖wrappedDek, so a KEK rewrap (new wrapped DEK, same revision) correctly marks
|
||||
sites out of date. `ComputeCurrentArtifactHashesAsync` hashes missing names with `Missing = true`, so
|
||||
a site referencing a not-yet-created secret reads *Out of date*, never *Current*. The list is
|
||||
null-when-empty and the serializer omits nulls, so **sites with no secret references keep their
|
||||
current hash** — only sites that reference secrets flip to *Out of date* once after rollout.
|
||||
|
||||
### 3.5 Site apply — both nodes, via a replicated table
|
||||
|
||||
The artifact apply runs on the active node's `DeploymentManagerActor` singleton, but `secrets.db` is
|
||||
a per-node file outside LocalDb. So:
|
||||
|
||||
1. **`deployed_secrets`** — a new LocalDb table in `SiteStorageSchema` (`name` PK, `row_json` = the
|
||||
`SecretArtifact` ciphertext row as JSON, `revision`, `updated_utc`, `is_removed` 0/1,
|
||||
`deployment_id`, `applied_at`), added to `SiteLocalDbSetup.ReplicatedTables` (becomes the 11th;
|
||||
LocalDb 0.3.x snapshots just that table on an already-replicating pair). No FKs (row 105 guard).
|
||||
Ciphertext only — the no-plaintext rule that kept `smtp_configurations` unregistered holds.
|
||||
`HandleDeployArtifacts` applies it full-set: upsert every shipped row, mark rows absent from the
|
||||
set `is_removed = 1` (kept, so the standby learns about the removal).
|
||||
2. **`DeployedSecretsMirror`** — a per-node service (every site node, not a singleton) that reconciles
|
||||
`deployed_secrets` into the node's own `ISecretStore`: `ApplyReplicatedAsync` for live rows,
|
||||
`DeleteAsync` for removed rows **only if the name was previously mirrored** (a local, unreplicated
|
||||
`mirrored_secrets` table records names this mirror wrote — an out-of-band secret of the same name
|
||||
that was never deployed is never deleted). After applying, it **trial-resolves** each deployed
|
||||
secret through `ISecretResolver`; a decrypt failure (KEK mismatch) is an error naming the secret
|
||||
and the row's `KekId`. It runs at startup, every `SiteRuntime:DeployedSecretsMirrorInterval`
|
||||
(default 15 s — the standby's convergence path), and on demand.
|
||||
3. The active node's apply calls the mirror **synchronously** after writing the table and fails the
|
||||
deploy response if any secret did not apply or verify; on success the response carries
|
||||
`SecretsApplied = n`. The standby converges through LocalDb replication + its own mirror within one
|
||||
interval.
|
||||
|
||||
### 3.6 Mixed versions
|
||||
|
||||
A pre-feature site ignores proto field 9 and answers `Success` with `secrets_applied` absent. Central
|
||||
treats *command carried ≥ 1 secret AND `SecretsApplied` is null* as a **failure** for that site
|
||||
(`site build predates secret deployment — upgrade the site`) and does not stamp the hash. A command
|
||||
with no secrets is unaffected, so an old site with no secret references deploys exactly as before.
|
||||
Upgrade central first, as always.
|
||||
|
||||
### 3.7 Logging and audit (U3)
|
||||
|
||||
- **Central, per site:** Information `Deploying {Count} secret(s) to site {SiteId}: {Name}@r{Revision},…`;
|
||||
Error on a missing secret (names + referencing connections). Never values or ciphertext.
|
||||
- **Site, active apply and every mirror pass that changes something:** Information per secret
|
||||
`Applied deployed secret {Name} r{Revision} (kek {KekId})` / `Removed deployed secret {Name}`;
|
||||
Error on apply/verify failure; one Information summary per pass. An in-sync mirror pass logs at
|
||||
Debug only.
|
||||
- **Audit:** one `IAuditService` row per site deploy that carried secrets — action `DeploySiteSecrets`,
|
||||
entity type `SiteSecrets`, entity id = site identifier, after-state `{ deploymentId, secrets:
|
||||
[{name, revision}], removed: [name], success }`. Same config-change audit as the existing
|
||||
`DeployArtifacts` row.
|
||||
- **Hub sweep (library `ZB.MOM.WW.Secrets` 0.6.3, scadaproj):** every sweep outcome is observable —
|
||||
in-sync sweeps at Debug plus an Information heartbeat at most once per `InSyncLogInterval`
|
||||
(default 1 h), a `Recovered after {Failures} failed sweep(s)` Information line, and a public
|
||||
`ISecretsHubSyncStatus` (`LastSweepUtc`, `LastSuccessUtc`, `ConsecutiveFailures`, `LastError`).
|
||||
Useful for the rig until the hub is retired and for any other hub consumer.
|
||||
|
||||
### 3.8 Hub retirement (U2)
|
||||
|
||||
Once the artifact path is live on the rig, ScadaBridge stops composing the hub: the Site role no
|
||||
longer registers `AddZbSecretsGrpcHubClient`, the Central role no longer maps the hub endpoint, and
|
||||
`StartupValidator` refuses `Secrets:Replication:Enabled=true` with `Mode=Grpc` with a message pointing
|
||||
at the artifact deploy. The library keeps the hub (other consumers); only ScadaBridge stops using it.
|
||||
The rig's site-a moves from the hub to the artifact path; central keeps the shared SQL store through
|
||||
`Secrets:Store:Provider=SqlServer`.
|
||||
|
||||
## 4. Rollout (deployment records, not code)
|
||||
|
||||
vd03 and QA need, per environment: the `zbsecrets` store in the configuration SQL Server + both
|
||||
centrals on `Secrets:Store:Provider=SqlServer`; **one KEK across every central and site node**
|
||||
(verify before enabling — the mirror's verify step reports a mismatch by name); existing hand-seeded
|
||||
site secrets recreated at central; then *Deploy Artifacts*. Tracked as a deferred-work register row.
|
||||
|
||||
## 5. Out of scope
|
||||
|
||||
Per-site explicit secret lists; secrets for external systems / database connections (they ship
|
||||
literal values today); plaintext transport; moving bootstrap secrets through the deploy.
|
||||
|
||||
## 6. Implementation notes (as built)
|
||||
|
||||
Execution (tasks 1–11, ledger `.superpowers/sdd/2026-09-28-site-secrets-artifact-deploy/progress.md`)
|
||||
matched this design in shape but landed several deviations, called out here so the doc stays
|
||||
accurate as the spec:
|
||||
|
||||
- **Central collection.** The scanner and per-site resolve described in §3.1/§3.3 are implemented
|
||||
as `SiteSecretsCollector` (`ZB.MOM.WW.ScadaBridge.DeploymentManager`), not inline in
|
||||
`ArtifactDeploymentService`. Fail-closed (§3.3) is per site, not per batch: a site with an
|
||||
unresolved reference is skipped (`_communicationService` is never called for it) while every
|
||||
other site in the same sweep proceeds normally.
|
||||
- **Audit (§3.7).** The `DeploySiteSecrets` / `SiteSecrets` audit row is written on **both**
|
||||
`DeployCoreAsync` (the sweep) and `RetryForSiteAsync` (the per-site retry), including the
|
||||
fail-closed path on each — the design's §3.7 wording ("one row per site deploy that carried
|
||||
secrets") did not call out the retry path explicitly; it is in scope and covered.
|
||||
- **Stamp gating (§3.6).** Implemented as designed, plus one addition: a **partial apply**
|
||||
(`SecretsApplied` less than the shipped count, not just the mixed-version `null` case) is also
|
||||
treated as a failure and is not stamped, with its own message (`Site applied {n} of {m}
|
||||
secrets`) distinct from the "predates secret deployment" wording.
|
||||
- **Hash (§3.4).** Implemented as designed. Confirmed explicitly (not just implied by
|
||||
"null-when-empty"): an **empty** `Secrets`/`MissingSecrets` list and a **null** one hash
|
||||
identically, and the pre-existing golden hash for a site with no secret references is
|
||||
unchanged by this feature (pinned by a dedicated regression test).
|
||||
- **Site apply (§3.5).** `HandleDeployArtifacts` writes the full `deployed_secrets` set and then
|
||||
runs `DeployedSecretsMirror.ReconcileAsync` **synchronously** before replying, exactly as
|
||||
designed. `SecretsApplied` counts only the names *this deploy shipped* that applied and
|
||||
verified without error — a mirror failure for a name this deploy did not ship (e.g. another,
|
||||
already-broken deployed secret, or an unrelated removal) is logged at Warning, excluded from
|
||||
the count, and does not fail the deploy. Removal failures are likewise Warnings, never deploy
|
||||
failures.
|
||||
- **`DeployedSecretsMirror` — apply is confirm-then-ledger, not the blind `ApplyReplicatedAsync`
|
||||
§3.5 originally described.** The real library store is strict last-writer-wins on
|
||||
`(UpdatedUtc, Revision)` (a tie is a no-op) and tombstones rather than hard-deletes, which the
|
||||
naive "apply then ledger" sequence gets wrong in two ways: a routine remove-then-re-add can
|
||||
no-op against a still-live tombstone while the ledger is written anyway (silently broken), and
|
||||
an out-of-band local secret with a genuinely newer stamp can be ledgered and later deleted by a
|
||||
central tombstone (destroying the operator's own secret). The mirror now always reads the row
|
||||
back after applying and only ledgers when the local ciphertext matches what was shipped; a
|
||||
tombstoned or ledger-owned row is safely re-stamped (`UpdatedUtc`/`Revision` bumped past the
|
||||
local value, including across clock skew) and re-applied, while a live row with different
|
||||
ciphertext that the ledger does **not** already own is left completely untouched and reported
|
||||
as a distinct out-of-band-conflict failure. Ownership is decided from `mirrored_secrets`
|
||||
membership at any revision, not from tombstone state alone — so a mirror-owned name that drifts
|
||||
locally (an operator's edit over an already-deployed name) is re-stamped back to the deployed
|
||||
value on the next pass: **central is authoritative for every name it deploys.** Every apply/
|
||||
delete also invalidates the shared `ISecretResolver` cache before the verify step, and one
|
||||
poisoned row never blocks the rest of the pass.
|
||||
- **Hub retirement (§3.8) — scope widened to ALL secrets replication, not just the gRPC hub.**
|
||||
ScadaBridge also retires the historical SQL-Server *replication* mode
|
||||
(`Secrets:Replication:Mode=SqlServer`, distinct from the still-supported
|
||||
`Secrets:Store:Provider=SqlServer` shared-store feature) for the same reason the gRPC hub goes:
|
||||
a site's `ISecretStore` must be the plain, undecorated local SQLite store, because
|
||||
`DeployedSecretsMirror` calls `DeleteAsync` directly and a replication decorator would publish
|
||||
that delete fleet-wide. `Secrets:Replication:Enabled` set to anything other than
|
||||
absent/empty/`false` (a deliberately strict, non-boolean-coerced check — `"0"`/`"1"`/`"yes"`
|
||||
all count as "set") now fails `StartupValidator.Validate` on **either** role. A leftover
|
||||
`Secrets:Replication:Mode` or any `Secrets:GrpcHub:*` key logs a startup Warning naming both
|
||||
sections. The pre-host `${secret:}` expander was fixed in the same task set to read the shared
|
||||
SQL store on a `Store:Provider=SqlServer` central (it previously only did this for the retired
|
||||
Grpc mode) — without that fix a central's boot-time `${secret:}` tokens and its running
|
||||
`ISecretResolver` could resolve a token differently.
|
||||
- **`ZB.MOM.WW.Secrets` 0.6.3 (hub sweep logging, §3.7's third bullet — `EventId`s 4/5/6,
|
||||
`ISecretsHubSyncStatus`) was implemented and tested on a `scadaproj` branch but is NOT
|
||||
published to the package feed, and ScadaBridge does not consume it.** Publishing is an outward
|
||||
side effect that needs explicit user consent, and it would be a dead pin here regardless —
|
||||
hub retirement (above) removes ScadaBridge's only dependency on the hub client entirely. It
|
||||
remains available for the library's other hub consumers once published.
|
||||
- **Empty, not null, when a site references no secrets (Task 13 live-gate fix).** The command
|
||||
builder first shipped `Secrets = null` for a site with no references ("null-when-empty", borrowed
|
||||
from the hash convention). But the site treats `null` as "central shipped no secrets" and touches
|
||||
nothing, so dropping a site's LAST `secret:` reference never soft-removed the row and the secret
|
||||
stayed live on both nodes. Central now always ships a list (empty when there are no references);
|
||||
the hash is unchanged because an empty and a null list hash identically.
|
||||
- **Final-review fixes.**
|
||||
- *Same-revision drift is healed.* A name already ledgered at the shipped revision is still read
|
||||
back every pass; a missing, tombstoned or edited local row is re-stamped back to the shipped
|
||||
ciphertext before verify (previously a local delete was misreported as a KEK mismatch and
|
||||
blocked the site's deploys).
|
||||
- *Unreadable `row_json`* is a per-name failure, not a pass-wide exception.
|
||||
- *Site ordering:* secrets are written and mirrored before the deploy's data connections are
|
||||
pushed to the DCL; a rotation with unchanged connection config takes effect on the connection's
|
||||
next (re)connect (no forced reconnect).
|
||||
- *One `secret:` parser* (`SecretReferenceScanner.TryParseSecretReference`) shared by the scanner
|
||||
and `MxGatewayDataConnection`, so a padded/case-variant reference resolves the shipped name.
|
||||
- *Audit shape (§3.7 deviation).* The `DeploySiteSecrets` row carries `secrets` and `missing` —
|
||||
there is **no `removed` field** — and a removal-only deploy (empty list shipped) writes **no
|
||||
central row**; removals are logged on the site by `DeployedSecretsMirror`.
|
||||
- *Deployment prerequisite:* a site pair using deployed secrets must run LocalDb replication
|
||||
(topology guide), since on an unreplicated pair only the node active at deploy time gets them.
|
||||
- **Rig posture.** `docker/docker-compose.yml`'s central nodes now carry
|
||||
`Secrets__Store__Provider=SqlServer` plus a shared KEK; site-a shares that KEK (it is the
|
||||
secrets-via-artifact-deploy site); site-b/site-c deliberately carry a **mismatched** KEK so the
|
||||
`DeployedSecretsMirror` verify-failure path has a real, provable negative on the rig rather than
|
||||
an assumed one.
|
||||
@@ -0,0 +1,153 @@
|
||||
# Site secrets ride the artifact deploy — live gate (Task 13)
|
||||
|
||||
**Date:** 2026-09-28/29 (UTC 00:28–00:50 on 2026-09-29) · **Branch:** `ScadaBridge-site-secrets` worktree ·
|
||||
**Result:** `bash docker/verify-site-secrets.sh` → **PASS 8/8** (on the image carrying fix `ad19d4b1`).
|
||||
The first run, on the pre-fix image, was **PASS 7/8** — check 5 found a product defect, fixed below.
|
||||
|
||||
Design: [`2026-09-28-site-secrets-artifact-deploy-design.md`](2026-09-28-site-secrets-artifact-deploy-design.md).
|
||||
|
||||
## 1. Method
|
||||
|
||||
### Rig bring-up
|
||||
|
||||
`docker/deploy.sh` was deliberately **not** run from the worktree (its relative bind mounts would rehome
|
||||
to the worktree). Instead:
|
||||
|
||||
1. `bash docker/build.sh` from the worktree → `scadabridge:latest` built from branch code.
|
||||
2. `docker compose -f <worktree>/docker/docker-compose.yml --project-directory ~/Desktop/ScadaBridge/docker up -d --force-recreate`
|
||||
— the branch compose (new `Secrets__*` env: central `Store:Provider=SqlServer` + shared KEK, site-a shared KEK,
|
||||
site-b/site-c a deliberately mismatched KEK, no hub keys anywhere) with bind mounts resolved against the primary
|
||||
checkout's `docker/` directory. `git diff 2aede140 HEAD --stat -- docker/` shows only `docker-compose.yml` changed,
|
||||
so the mounted node appsettings are identical to the branch's. `docker compose config` resolved project name
|
||||
`docker`, the running project, so the containers were replaced, not duplicated.
|
||||
3. All 8 nodes answered `/health/ready` 200 (centrals on 9001/9002; sites probed on their internal 8084 from the
|
||||
Traefik container, which is on `scadabridge-net`).
|
||||
|
||||
The same procedure was repeated after the fix (image rebuild + force-recreate); all 8 ready within ~10 s.
|
||||
|
||||
### Boot-log review (secrets)
|
||||
|
||||
- **Central (both):** no secrets-related warning or error. `Secrets:Store:Provider=SqlServer` against the rig's
|
||||
existing `ZbSecretsHub` database came up clean; no leftover-key warning (the rig carries no `Secrets:GrpcHub:*` or
|
||||
`Replication:Mode`).
|
||||
- **site-a:** no secrets-related warning or error. Its local `secrets.db` still holds six rows swept from the retired
|
||||
hub in August (`hub-gate-smoke-2/3`, `shared-gate-*`, KEK `sha256:f94d4ae2c4b1`) — inert, not ledgered by the
|
||||
mirror, so never touched by it.
|
||||
- **site-b / site-c:** booted cleanly with a KEK for the first time; no secrets warnings.
|
||||
- The only secrets line after the second recreate was an expected `ERR` on site-b-a — the mirror's startup pass
|
||||
re-verifying the `sb-gate-mxgw` row left live there by the pre-fix run (the defect's residue); the next gate
|
||||
run's pre-clean removed it.
|
||||
- Unrelated noise present on every boot: singleton-proxy "not found after 30 s" warnings during cluster formation,
|
||||
one `akka-cluster ... Removed` health-check ERR per pair during the recreate, OPC UA auto-accept warnings.
|
||||
|
||||
### How each surface is driven
|
||||
|
||||
| Surface | Tool | Notes |
|
||||
|---|---|---|
|
||||
| Central secret create / rotate / delete | Central UI **`/admin/secrets`** (ZB.MOM.WW.Secrets.Ui), headless Chromium via Python Playwright | The supported operator path; there is no CLI or management verb for central secrets, and the library's `secret` CLI only talks to a SQLite store headless. Value goes through the environment into the masked input, never argv/stdout. |
|
||||
| Data connections, artifact deploys, audit query | ScadaBridge CLI via Traefik (`localhost:9000`) | `data-connection create/update/delete`, `site deploy-artifacts --site-id`, `audit-config query --action DeploySiteSecrets`. |
|
||||
| Artifacts badge (Current / Out of date) | Sites page, **Re-check** pressed, badge `aria-label` read | The pending-updates evaluation has no CLI surface and is Blazor-interactive. |
|
||||
| Central row | `sqlcmd` **read-only** on `ZbSecretsHub.zbsecrets.secret` | Revision, KekId, `HASHBYTES('SHA2_256', ciphertext)`. |
|
||||
| Site node stores | `docker cp` of `/app/data/scadabridge-secrets.db` (library table `secret`) and `/app/data/site-localdb.db` (`deployed_secrets`, `mirrored_secrets`, `data_connection_definitions`) + WAL, read by host python `sqlite3` | The image has no sqlite3. The `-shm` is not copied (SQLite rebuilds the wal-index); each read is repeated until two consecutive copies agree. |
|
||||
| Out-of-band seed on site-a-a (check 5) | The library's own **`secret` CLI** (`ZB.MOM.WW.Secrets.Cli`, published from scadaproj main = 0.6.2) run in a throwaway `scadabridge:latest` container with `--volumes-from scadabridge-site-a-a`, `Secrets__SqlitePath=/data/scadabridge-secrets.db` and site-a's KEK | i.e. the library's normal local SQLite write path on that node's store — not a raw SQL insert. |
|
||||
|
||||
Gate objects: central secret `sb-gate-mxgw`; never-created `sb-gate-missing`; out-of-band `sb-gate-oob` on site-a-a;
|
||||
MxGateway connections `sb-gate-conn-a` (site-a) and `sb-gate-conn-b` (site-b), unbound, endpoint
|
||||
`http://sb-gate-no-such-gateway:5000`. The trap-based teardown (also run up-front as a pre-clean) deletes both
|
||||
connections, redeploys site-a and site-b so the removal reaches them, deletes the central secret, removes the
|
||||
out-of-band secret with the `secret` CLI, then redeploys artifacts fleet-wide; the run ended with all three sites
|
||||
**Current**. The library tombstones rather than hard-deletes, so after teardown the stores hold only
|
||||
`is_deleted = 1` rows for the gate names (verified: site-a-a, site-a-b, site-b-a and central all `deleted`;
|
||||
no gate connection left).
|
||||
|
||||
## 2. Per-check evidence (run 2, PASS 8/8)
|
||||
|
||||
| # | Check | Evidence |
|
||||
|---|---|---|
|
||||
| 1 | Secret + referencing connection → site-a Out of date, b/c unchanged | `sb-gate-mxgw` created via `/admin/secrets` (central r3, KEK `sha256:f94d4ae2c4b1` — the revision continues past run 1's tombstone); after `sb-gate-conn-a` (`"ApiKey":"secret:sb-gate-mxgw"`) badges `site-a: out of date, site-b: current, site-c: current`. |
|
||||
| 2 | Deploy site-a → success, `secretsDeployed: 1`, Current | `{"siteId":"site-a","success":true,"secretsDeployed":1}` (0.2 s); badge `current`. |
|
||||
| 3 | Both nodes at the central revision; standby ≤ 30 s | site-a-a r3, ciphertext SHA-256 `707ae73ba872…` == central (visible 4.0 s after the deploy returned — dominated by the docker-cp read loop; the active node applies synchronously before replying); **site-a-b converged 5.5 s** after the deploy returned; its `mirrored_secrets` ledger at r3. |
|
||||
| 4 | Rotate → Out of date → deploy → both at new revision | central r3 → r4 with new ciphertext; badge `out of date`; redeploy `secretsDeployed 1`; site-a-a r4 (0.3 s), **site-a-b r4 at 6.8 s**; badge back to `current`. |
|
||||
| 5 | Drop the reference → both nodes drop it; out-of-band survives | `sb-gate-oob` seeded on site-a-a via the `secret` CLI; `sb-gate-conn-a` switched to a literal ApiKey; deploy succeeded (no `secretsDeployed`); site-a-a no longer holds `sb-gate-mxgw` (0.2 s), **site-a-b at 9.0 s**; `sb-gate-oob` still live on site-a-a; `deployed_secrets.sb-gate-mxgw.is_removed = 1` (kept for the standby). |
|
||||
| 6 | Missing secret → fail closed, stays Out of date, site untouched | Error `Secret 'sb-gate-missing' referenced by data connection(s) sb-gate-conn-a is not in the central secret store`; badge `out of date`; stamped hash, both nodes' secret rows, `deployed_secrets`, ledgers and the site's `sb-gate-conn-a` definition byte-identical before/after. |
|
||||
| 7 | Same reference on site-b (mismatched KEK) → fails naming KekId | Error `Secret 'sb-gate-mxgw' (kek sha256:f94d4ae2c4b1) did not decrypt on this node — the node's master key differs from central's`; site-b not stamped (`out of date`). |
|
||||
| 8 | Audit rows; no material in logs | Exactly 4 `DeploySiteSecrets`/`SiteSecrets` rows with distinct deploymentIds: `site-a ok sb-gate-mxgw@r3`, `site-a ok @r4`, `site-a fail missing sb-gate-missing`, `site-b fail @r4`; the no-secret deploy of check 5 wrote none. Grep of `docker logs --since <gate start>` on all 8 nodes and of the audit payloads for both plaintext values, the out-of-band value and both base64 ciphertexts: **0 hits**. |
|
||||
|
||||
### Standby convergence (site-a-b, seconds after the deploy call returned)
|
||||
|
||||
| Run | Check 3 (first deploy) | Check 4 (rotation) | Check 5 (removal) |
|
||||
|---|---|---|---|
|
||||
| 1 (pre-fix) | 9.4 | 5.4 | never (defect) |
|
||||
| 2 (fixed) | 5.5 | 6.8 | 9.0 |
|
||||
|
||||
All inside the 30 s window, consistent with LocalDb CDC replication of `deployed_secrets` plus the standby's
|
||||
15 s `DeployedSecretsMirrorInterval`. The figures include up to ~1–2 s of read-loop overhead (docker cp + double
|
||||
read per poll).
|
||||
|
||||
## 3. Defects found
|
||||
|
||||
### 3.1 Dropping a site's last `secret:` reference never removed the secret (fixed, `ad19d4b1`)
|
||||
|
||||
Run 1, check 5: after the removal deploy both site-a nodes still held `sb-gate-mxgw` r1 live,
|
||||
`deployed_secrets.is_removed = 0`, ledger r1. Cause: `ArtifactDeploymentService.BuildDeployArtifactsCommandAsync`
|
||||
shipped `Secrets = null` whenever a site had no references ("null-when-empty"), and the site deliberately treats
|
||||
`null` as "central shipped no secrets" (pre-feature contract) and touches nothing — so the full-set soft-remove
|
||||
never ran. Fix: a secrets-aware central always ships a list (empty when there are no references). Old sites
|
||||
ignore proto field 9; the artifact hash is unchanged (empty and null hash identically); `SecretsDeployed` stays
|
||||
null and no audit row is written when nothing shipped. Regression test
|
||||
`NoSecretReferences_ShipsEmptySecretsList_SoTheSiteRemovesPreviouslyDeployedOnes` (red on the old code, green on
|
||||
the fix). Spec updated in `Component-DeploymentManager.md`, `Component-SiteRuntime.md` and the design doc's
|
||||
as-built notes.
|
||||
|
||||
### 3.2 `ManagementActorTests` did not compile a valid proxy (fixed, `1444f859`)
|
||||
|
||||
Four `DeployArtifacts_*` tests built an NSubstitute proxy of `ArtifactDeploymentService` with the pre-Task-6
|
||||
constructor arguments; they now pass a `SiteSecretsCollector`. Test-only.
|
||||
|
||||
## 4. Test suite
|
||||
|
||||
`dotnet build ZB.MOM.WW.ScadaBridge.slnx`: 0 warnings, 0 errors.
|
||||
`dotnet test ZB.MOM.WW.ScadaBridge.slnx` (30 test assemblies, run against the live rig): **10 006 tests, 9 979
|
||||
passed, 27 failed** — 4 were the `ManagementActorTests` proxy failures fixed in `1444f859`
|
||||
(ManagementService.Tests re-run: 668/668), the other 23 are Playwright E2E. After the fix commit
|
||||
DeploymentManager.Tests is 234/234 (one new test). Every non-Playwright assembly is green.
|
||||
|
||||
Playwright residue (23 of 184):
|
||||
|
||||
| Class | Tests | Classification |
|
||||
|---|---|---|
|
||||
| `RoleNavigationTests` | `AdminUser_DoesNotSeeDeploymentSection`, `AdminUser_DoesNotSeeDesignSection`, `AdminUser_SeesHealthDashboard_NotDeploymentMonitoring`, `DeploymentUser_DoesNotSeeAdminSection`, `DeploymentUser_DoesNotSeeDesignSection`, `DeploymentUser_SeesMonitoringButNotConfigurationAuditLog`, `DesignUser_DoesNotSeeAdminSection`, `DesignUser_DoesNotSeeDeploymentSection`, `DesignUser_SeesHealthDashboard_NotDeploymentMonitoringOrAudit` | Known env-only — role-negative under `DisableLogin=true` |
|
||||
| `LoginTests` | `UnauthenticatedUser_RedirectsToLogin` | Known env-only — `DisableLogin` |
|
||||
| `AuditLogPageTests` | `PermissionGating_DesignerWithoutOperationalAudit_IsDeniedAccess` | Known env-only — `DisableLogin` |
|
||||
| `DeploymentActionTests` | `Delete_Instance_RemovesFromTree`, `Deploy_Instance_ShowsOutcomeToast`, `Disable_Instance_ShowsOutcomeToast`, `Enable_Instance_ShowsOutcomeToast` | Known residue (pagination helper — row not found) |
|
||||
| `TopologyAreaTests` | `CreateArea_ViaToolbar_AppearsInTreeAndPersists`, `Diff_OnDeployedInstance_OpensDialog`, `MoveArea_UnderAnotherArea_ShowsMovedToast`, `MoveInstance_ToArea_ShowsMovedToast`, `RenameArea_EnterCommits_PersistsNewName`, `RenameArea_Escape_RevertsToOriginalLabel` | Known residue (SignalR/visibility timeout) |
|
||||
| `AuditLogPageTests` | `ExportCsv_LinkVisibleForAuditExportUserAndTriggersDownload` | **Not on the known list** — `Download` event timeout (30 s), reproducible in isolation. Audit export is untouched by this branch; central logs showed audit-ingest SQL timeouts against the large `AuditLog` table at the time. Judged environmental. |
|
||||
| `TransportImportTests` | `ImportSyntheticBundle_AppliesAndShowsAuditDrillIn` | **Not on the known list** — reproducible in isolation; central logs `Cannot insert duplicate key row in object 'dbo.Sites' ... (site-a)` during the import, i.e. the synthetic bundle's site mapping collides with the rig's existing site-a. Transport and site mapping are untouched by this branch. Judged rig-data related. |
|
||||
|
||||
## 5. Observations (not gated)
|
||||
|
||||
- **Standby of an unreplicated site pair never receives deployed secrets.** `deployed_secrets` reaches the
|
||||
standby only through LocalDb CDC replication; on site-b (deliberately unreplicated on this rig) the gate secret
|
||||
landed in site-b-a's store only — site-b-b's store never saw it. A failover on an unreplicated pair would leave
|
||||
the new active node without the site's secrets until the next artifact deploy. This follows from the design
|
||||
(§3.5) but is worth stating in the topology guide: secrets-bearing sites should run with LocalDb replication on.
|
||||
- The CLI's `site deploy-artifacts` exits 0 even when a site result is `success: false`; the gate reads the JSON
|
||||
`siteResults` rather than the exit code.
|
||||
- Re-adding a name that is tombstoned at central continues its revision (run 2 started at r3 after run 1's r1 +
|
||||
delete), which is why the gate compares revisions to the live central row rather than to fixed numbers.
|
||||
|
||||
## 6. Re-run after final-review fixes (2026-09-28)
|
||||
|
||||
Code under test: `d6411863` (fixes `6f1787ab` mirror drift heal + unreadable-row isolation, `b1b49bb3`
|
||||
secrets-before-DCL ordering, `67a3950f` shared `secret:` parser + hash-parity test). Same procedure as §1:
|
||||
`bash docker/build.sh` from the worktree (image `sha256:9de1db5c…`), then
|
||||
`docker compose -f <worktree>/docker/docker-compose.yml --project-directory ~/Desktop/ScadaBridge/docker up -d
|
||||
--force-recreate`; all 8 nodes answered `/health/ready` 200 within ~15 s.
|
||||
|
||||
`bash docker/verify-site-secrets.sh` → **PASS 8/8**. Standby convergence on site-a-b (seconds after the deploy
|
||||
call returned): check 3 = 8.1, check 4 = 5.4, check 5 = 9.0. Check 8: exactly the 4 expected `DeploySiteSecrets`
|
||||
rows, 0 plaintext/ciphertext hits across all 8 node logs and the audit payloads. Teardown left all three sites
|
||||
*Current*.
|
||||
|
||||
Tests before the rig run: `dotnet build ZB.MOM.WW.ScadaBridge.slnx` 0 errors; Commons.Tests 875/875,
|
||||
SiteRuntime.Tests 892/892, DeploymentManager.Tests 235/235, DataConnectionLayer.Tests 383/383, Host.Tests 593/593.
|
||||
@@ -0,0 +1,989 @@
|
||||
# Site Secrets via Artifact Deploy — Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:subagent-driven-development (recommended) or superpowers-extended-cc:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Ship each site exactly the secrets its data connections reference as part of the on-demand artifact deploy, count them in the site's artifact staleness hash, log and audit every secret deployment, and retire the gRPC secrets hub for ScadaBridge sites.
|
||||
|
||||
**Architecture:** Central scans the site's data-connection configs for `secret:<name>` references, reads those rows (ciphertext, never plaintext) from one shared SQL-Server secret store, and adds them to `DeployArtifactsCommand` and to `ArtifactRevisionHashService`. The site's active node writes them into a new LocalDb-replicated `deployed_secrets` table. A per-node `DeployedSecretsMirror` then applies them into that node's own `secrets.db` with `ApplyReplicatedAsync` and checks each one decrypts. The standby converges through LocalDb replication plus its own mirror.
|
||||
|
||||
**Tech Stack:** .NET 10, Akka.NET, gRPC (vendored generated code), SQLite via ZB.MOM.WW.LocalDb 0.3.4, ZB.MOM.WW.Secrets 0.6.2 → 0.6.3, xUnit, docker rig.
|
||||
|
||||
**Spec:** `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md` (read it first — every section number cited below refers to it).
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- **Never plaintext on the wire, in LocalDb, in logs or in audit rows.** Only `StoredSecret` ciphertext fields cross central→site; logs/audit carry names, revisions, KekIds, counts.
|
||||
- **Additive contracts only:** new proto fields `DeployArtifactsCommandDto.secrets = 9`, `ArtifactDeploymentResponseDto.secrets_applied` (next free field number, `google.protobuf.Int32Value`); new record parameters appended with `= null` defaults. Vendored generated C# regenerated with `bash docker/regen-proto.sh sitecommand`; an active `<Protobuf>` item must never be committed.
|
||||
- **Hash determinism:** every `Hashable*` record declares properties alphabetically (guarded by `ArtifactRevisionHashServiceTests.HashableRecords_PropertiesDeclaredAlphabetically`); lists sorted by name ordinal; null-when-empty.
|
||||
- **Bootstrap secrets are never deployed** (gRPC PSK `SB-GRPC-PSK-*`, LocalDb replication key, master key) — design §3.1.
|
||||
- **Fail closed:** a referenced secret missing/tombstoned at central fails that site's deploy before sending; a secret that fails to apply or decrypt on the active site node fails the deploy response.
|
||||
- **Never `await` an audit write on a script path** (not touched here, but do not introduce one).
|
||||
- **Do not use `git stash`** (shared across worktrees). Commit with pathspecs.
|
||||
- Build `dotnet build ZB.MOM.WW.ScadaBridge.slnx`; per-task run the filtered tests named in each task; full solution test + rig rebuild only in Task 13.
|
||||
|
||||
**User decisions (already made):**
|
||||
- "Which secrets should a site receive with its artifacts?" → **Only referenced** (derived from the site's data-connection `secret:` refs).
|
||||
- "How should this relate to the existing gRPC hub pull?" → **Replace it.**
|
||||
- "What should the logging cover?" → **Deploy log lines, Audit rows, Hub sweep logging.**
|
||||
- "Central needs one authoritative secret store to deploy from" → **Shared SQL store.**
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
| File | Responsibility |
|
||||
|---|---|
|
||||
| `~/Desktop/scadaproj/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.Grpc/GrpcSecretSyncService.cs` (+ new `ISecretsHubSyncStatus.cs`, `SecretsHubSyncStatus.cs`) | Library 0.6.3: sweep outcome logging + public status |
|
||||
| `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Artifacts/SecretArtifact.cs` (new) | Wire DTO for one ciphertext secret row |
|
||||
| `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Artifacts/DeployArtifactsCommand.cs`, `ArtifactDeploymentResponse.cs` | Additive `Secrets` / `SecretsApplied` |
|
||||
| `src/ZB.MOM.WW.ScadaBridge.Communication/Protos/site_command.proto`, `Communication/SiteCommandGrpc/*` (vendored), `Communication/Grpc/SiteCommandDtoMapper.cs` | Wire mapping |
|
||||
| `src/ZB.MOM.WW.ScadaBridge.Commons/Types/Secrets/SecretReferenceScanner.cs` (new) | Derive referenced secret names from data-connection JSON |
|
||||
| `src/ZB.MOM.WW.ScadaBridge.DeploymentManager/ArtifactRevisionHashService.cs` | Secrets in the staleness hash |
|
||||
| `src/ZB.MOM.WW.ScadaBridge.DeploymentManager/SiteSecretsCollector.cs` (new) | Central: scan + load rows from `ISecretStore` + map to `SecretArtifact` |
|
||||
| `src/ZB.MOM.WW.ScadaBridge.DeploymentManager/ArtifactDeploymentService.cs` | Build/fail-closed/stamp gating/audit/logging |
|
||||
| `src/ZB.MOM.WW.ScadaBridge.Host/SecretsRegistration.cs`, `StartupValidator.cs` | `Secrets:Store:Provider`, hub retirement, warnings |
|
||||
| `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageSchema.cs`, `SiteStorageService.cs` | `deployed_secrets` + `mirrored_secrets` tables and accessors |
|
||||
| `src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs` | Register `deployed_secrets` as replicated table 11 |
|
||||
| `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Secrets/DeployedSecretsMirror.cs` (new) | Per-node reconcile `deployed_secrets` → `secrets.db` + verify |
|
||||
| `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs` | Apply secrets in `HandleDeployArtifacts` |
|
||||
| `docker/docker-compose.yml`, `docker/verify-site-secrets.sh` (new) | Rig switch-over + live gate |
|
||||
| docs (`Component-*.md`, CLAUDE.md, register, umbrella index) | Spec sync |
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Secrets library 0.6.3 — hub sweep logging + status
|
||||
|
||||
**Goal:** Every hub sweep outcome is observable (Debug in-sync + hourly Information heartbeat + recovery line) and a public `ISecretsHubSyncStatus` exposes last sweep/success/failures; published as 0.6.3 and pinned in ScadaBridge.
|
||||
|
||||
**Files:**
|
||||
- Modify: `~/Desktop/scadaproj/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.Grpc/GrpcSecretSyncService.cs`
|
||||
- Create: `~/Desktop/scadaproj/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.Grpc/ISecretsHubSyncStatus.cs`
|
||||
- Modify: `~/Desktop/scadaproj/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.Grpc/SecretsGrpcHubClientOptions.cs` (add `InSyncLogInterval`, default 1 h, validated > 0)
|
||||
- Modify: the Grpc hub client DI extension (register `ISecretsHubSyncStatus` singleton)
|
||||
- Modify: `~/Desktop/scadaproj/ZB.MOM.WW.Secrets/Directory.Build.props` (`<Version>0.6.3</Version>`), `~/Desktop/scadaproj/ZB.MOM.WW.Secrets/CLAUDE.md` (version narrative)
|
||||
- Test: `~/Desktop/scadaproj/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.Grpc.Tests/GrpcSecretSyncServiceLoggingTests.cs` (new)
|
||||
- Modify: `Directory.Packages.props` (ScadaBridge — all `ZB.MOM.WW.Secrets*` to 0.6.3)
|
||||
|
||||
**Acceptance Criteria:**
|
||||
- [ ] An in-sync sweep (0 pulled) logs EventId 4 at Debug every time and EventId 5 `Secrets hub in sync ({Count} secret(s)); last change pulled {LastPullUtc}` at Information at most once per `InSyncLogInterval`.
|
||||
- [ ] The first successful sweep after ≥ 1 failure logs EventId 6 `Secrets hub sync recovered after {Failures} failed sweep(s)` at Information.
|
||||
- [ ] `ISecretsHubSyncStatus` returns `LastSweepUtc`, `LastSuccessUtc`, `ConsecutiveFailures`, `LastError` (message only, never a secret value) reflecting the last sweep.
|
||||
- [ ] Existing EventIds 1–3 unchanged; library tests green; package 0.6.3 on the Gitea feed; ScadaBridge builds on 0.6.3.
|
||||
|
||||
**Verify:** `cd ~/Desktop/scadaproj/ZB.MOM.WW.Secrets && dotnet test --filter FullyQualifiedName~GrpcSecretSyncServiceLoggingTests` → all pass; `cd ~/Desktop/ScadaBridge && dotnet build ZB.MOM.WW.ScadaBridge.slnx` → 0 errors.
|
||||
|
||||
**Steps:**
|
||||
|
||||
- [ ] **Step 1: Write the failing tests** (fake hub reader returning an unchanged manifest; `FakeTimeProvider`; capture logs with a list logger):
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task InSyncSweep_LogsDebugEveryTime_AndInformationHeartbeatOncePerInterval()
|
||||
{
|
||||
var time = new FakeTimeProvider(DateTimeOffset.Parse("2026-09-28T00:00:00Z"));
|
||||
var (svc, logs, _) = Harness.Create(time, inSyncLogInterval: TimeSpan.FromHours(1), manifestUnchanged: true);
|
||||
await svc.SweepForTestAsync(CancellationToken.None);
|
||||
time.Advance(TimeSpan.FromMinutes(30));
|
||||
await svc.SweepForTestAsync(CancellationToken.None);
|
||||
time.Advance(TimeSpan.FromMinutes(31));
|
||||
await svc.SweepForTestAsync(CancellationToken.None);
|
||||
Assert.Equal(3, logs.Count(l => l.EventId.Id == 4 && l.Level == LogLevel.Debug));
|
||||
Assert.Equal(2, logs.Count(l => l.EventId.Id == 5 && l.Level == LogLevel.Information));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task FirstSuccessAfterFailures_LogsRecovered_AndStatusResets()
|
||||
{
|
||||
var time = new FakeTimeProvider();
|
||||
var (svc, logs, status) = Harness.Create(time, failFirst: 2);
|
||||
await svc.SweepForTestAsync(default); await svc.SweepForTestAsync(default);
|
||||
Assert.Equal(2, status.ConsecutiveFailures);
|
||||
await svc.SweepForTestAsync(default);
|
||||
Assert.Contains(logs, l => l.EventId.Id == 6 && l.Message.Contains("after 2 failed"));
|
||||
Assert.Equal(0, status.ConsecutiveFailures);
|
||||
Assert.NotNull(status.LastSuccessUtc);
|
||||
}
|
||||
```
|
||||
|
||||
(`SweepForTestAsync` is an `internal` wrapper over the existing `SweepAsync`; add `InternalsVisibleTo` for the test project if missing.)
|
||||
|
||||
- [ ] **Step 2: Run to verify they fail** — `dotnet test --filter FullyQualifiedName~GrpcSecretSyncServiceLoggingTests` → FAIL (EventIds 4–6 / status type missing).
|
||||
|
||||
- [ ] **Step 3: Implement.** Add the status type:
|
||||
|
||||
```csharp
|
||||
namespace ZB.MOM.WW.Secrets.Replicator.Grpc;
|
||||
|
||||
/// <summary>Point-in-time outcome of the hub follower's sweeps. Never carries secret material.</summary>
|
||||
public interface ISecretsHubSyncStatus
|
||||
{
|
||||
DateTimeOffset? LastSweepUtc { get; }
|
||||
DateTimeOffset? LastSuccessUtc { get; }
|
||||
int ConsecutiveFailures { get; }
|
||||
string? LastError { get; }
|
||||
}
|
||||
|
||||
internal sealed class SecretsHubSyncStatus : ISecretsHubSyncStatus
|
||||
{
|
||||
private readonly object _gate = new();
|
||||
public DateTimeOffset? LastSweepUtc { get; private set; }
|
||||
public DateTimeOffset? LastSuccessUtc { get; private set; }
|
||||
public int ConsecutiveFailures { get; private set; }
|
||||
public string? LastError { get; private set; }
|
||||
|
||||
internal int RecordSuccess(DateTimeOffset now)
|
||||
{
|
||||
lock (_gate) { var prior = ConsecutiveFailures; LastSweepUtc = LastSuccessUtc = now; ConsecutiveFailures = 0; LastError = null; return prior; }
|
||||
}
|
||||
|
||||
internal void RecordFailure(DateTimeOffset now, Exception ex)
|
||||
{
|
||||
lock (_gate) { LastSweepUtc = now; ConsecutiveFailures++; LastError = ex.GetType().Name + ": " + ex.Message; }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
In `GrpcSecretSyncService.SweepAsync`, after the reconcile succeeds: `var prior = _status.RecordSuccess(now)`; if `prior > 0` → `LogSweepRecovered(prior)`; if `pulled > 0` → existing `LogSweepConverged(pulled)`, record `_lastPullUtc = now`; else `LogSweepInSync(count)` (Debug) and, when `now - _lastInSyncInfo >= _options.InSyncLogInterval` or it has never logged, `LogSweepInSyncHeartbeat(count, _lastPullUtc)` and set `_lastInSyncInfo = now`. In the catch: `_status.RecordFailure(now, ex)` then the existing `LogSweepFailed(ex)`. New `[LoggerMessage]` EventIds 4 (Debug), 5 (Information), 6 (Information). Register `SecretsHubSyncStatus` as a singleton for both the concrete type and `ISecretsHubSyncStatus` in the hub-client DI extension, and inject it into the service.
|
||||
|
||||
- [ ] **Step 4: Run tests** → PASS; run the whole library suite `dotnet test` → PASS.
|
||||
|
||||
- [ ] **Step 5: Version, pack, push, pin.** Bump `<Version>` to 0.6.3, add the 0.6.3 entry to `ZB.MOM.WW.Secrets/CLAUDE.md`, run the repo's existing pack/push script for Secrets (see `ZB.MOM.WW.Secrets/CLAUDE.md` → publishing), commit in scadaproj. In ScadaBridge set every `ZB.MOM.WW.Secrets*` `PackageVersion` in `Directory.Packages.props` to `0.6.3`, `dotnet build ZB.MOM.WW.ScadaBridge.slnx`.
|
||||
|
||||
```bash
|
||||
git -C ~/Desktop/scadaproj add ZB.MOM.WW.Secrets && git -C ~/Desktop/scadaproj commit -m "feat(secrets): 0.6.3 — hub sweep outcome logging (in-sync debug + hourly heartbeat + recovered) and ISecretsHubSyncStatus"
|
||||
git add Directory.Packages.props && git commit -m "build: pin ZB.MOM.WW.Secrets 0.6.3 (hub sweep logging)"
|
||||
```
|
||||
|
||||
```json:metadata
|
||||
{"files": ["~/Desktop/scadaproj/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.Grpc/GrpcSecretSyncService.cs", "~/Desktop/scadaproj/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.Grpc/ISecretsHubSyncStatus.cs", "Directory.Packages.props"], "verifyCommand": "cd ~/Desktop/scadaproj/ZB.MOM.WW.Secrets && dotnet test --filter FullyQualifiedName~GrpcSecretSyncServiceLoggingTests", "acceptanceCriteria": ["EventId 4 Debug per in-sync sweep, EventId 5 Information at most once per InSyncLogInterval", "EventId 6 recovered line after failures", "ISecretsHubSyncStatus reflects last sweep", "0.6.3 published and pinned; ScadaBridge builds"], "modelTier": "standard"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Wire contract — `SecretArtifact`, `Secrets`, `SecretsApplied`
|
||||
|
||||
**Goal:** `DeployArtifactsCommand` carries an optional ciphertext secret list and `ArtifactDeploymentResponse` an optional applied count, end to end over gRPC, backward compatible.
|
||||
|
||||
**Files:**
|
||||
- Create: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Artifacts/SecretArtifact.cs`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Artifacts/DeployArtifactsCommand.cs`, `ArtifactDeploymentResponse.cs`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Communication/Protos/site_command.proto` (fields per Global Constraints), vendored `src/ZB.MOM.WW.ScadaBridge.Communication/SiteCommandGrpc/*` via `docker/regen-proto.sh sitecommand`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteCommandDtoMapper.cs` (`ToProto`/`FromProto` for command and response)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteCommandDtoMapperSecretsTests.cs` (new)
|
||||
|
||||
**Acceptance Criteria:**
|
||||
- [ ] Command round-trip preserves every `SecretArtifact` field byte-for-byte; `Secrets == null` round-trips as null (field absent), empty list as empty.
|
||||
- [ ] Response round-trip: `SecretsApplied` null ⇄ absent wrapper, `3` ⇄ `3`.
|
||||
- [ ] A `DeployArtifactsCommandDto` serialized WITHOUT field 9 (old central) maps to `Secrets == null`.
|
||||
- [ ] No `<Protobuf>` item left active in the csproj.
|
||||
|
||||
**Verify:** `dotnet test tests/ZB.MOM.WW.ScadaBridge.Communication.Tests --filter FullyQualifiedName~SiteCommandDtoMapperSecretsTests` → all pass.
|
||||
|
||||
**Steps:**
|
||||
|
||||
- [ ] **Step 1: Contracts.**
|
||||
|
||||
```csharp
|
||||
namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
|
||||
|
||||
/// <summary>
|
||||
/// One secret shipped to a site with its artifacts: the ZB.MOM.WW.Secrets StoredSecret row
|
||||
/// VERBATIM — ciphertext under the fleet's shared KEK, never plaintext. The site applies it
|
||||
/// with ISecretStore.ApplyReplicatedAsync. Design: docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md §3.3.
|
||||
/// </summary>
|
||||
public sealed record SecretArtifact(
|
||||
string Name,
|
||||
string ContentType,
|
||||
byte[] Ciphertext,
|
||||
byte[] Nonce,
|
||||
byte[] Tag,
|
||||
byte[] WrappedDek,
|
||||
byte[] WrapNonce,
|
||||
byte[] WrapTag,
|
||||
string KekId,
|
||||
long Revision,
|
||||
DateTimeOffset CreatedUtc,
|
||||
DateTimeOffset UpdatedUtc,
|
||||
string? Description = null);
|
||||
```
|
||||
|
||||
Append to `DeployArtifactsCommand`: `IReadOnlyList<SecretArtifact>? Secrets = null` (after `Timestamp`). Append to `ArtifactDeploymentResponse`: `int? SecretsApplied = null`.
|
||||
|
||||
- [ ] **Step 2: Proto.** Add `message SecretArtifactDto { string name = 1; string content_type = 2; bytes ciphertext = 3; bytes nonce = 4; bytes tag = 5; bytes wrapped_dek = 6; bytes wrap_nonce = 7; bytes wrap_tag = 8; string kek_id = 9; int64 revision = 10; google.protobuf.Timestamp created_utc = 11; google.protobuf.Timestamp updated_utc = 12; google.protobuf.StringValue description = 13; }`, `message SecretArtifactListDto { repeated SecretArtifactDto items = 1; }`, `SecretArtifactListDto secrets = 9; // absent => null` on `DeployArtifactsCommandDto`, and `google.protobuf.Int32Value secrets_applied = <next free>;` on `ArtifactDeploymentResponseDto`. Run `bash docker/regen-proto.sh sitecommand`; confirm `git diff --stat` shows only the vendored files + proto and the csproj unchanged.
|
||||
|
||||
- [ ] **Step 3: Failing tests** in `SiteCommandDtoMapperSecretsTests`:
|
||||
|
||||
```csharp
|
||||
private static SecretArtifact Sample(string name) => new(name, "text/plain",
|
||||
[1, 2, 3], [4], [5], [6, 7], [8], [9], "kek-A", 7,
|
||||
DateTimeOffset.Parse("2026-09-01T00:00:00Z"), DateTimeOffset.Parse("2026-09-02T00:00:00Z"), "d");
|
||||
|
||||
[Fact]
|
||||
public void DeployArtifacts_Secrets_RoundTrip_Verbatim()
|
||||
{
|
||||
var cmd = new DeployArtifactsCommand("dep", null, null, null, null, null, null,
|
||||
DateTimeOffset.UtcNow, [Sample("A"), Sample("B")]);
|
||||
var back = (DeployArtifactsCommand)SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(cmd));
|
||||
Assert.Equal(2, back.Secrets!.Count);
|
||||
Assert.Equal(cmd.Secrets![0].Ciphertext, back.Secrets[0].Ciphertext);
|
||||
Assert.Equal("kek-A", back.Secrets[1].KekId);
|
||||
Assert.Equal(7, back.Secrets[1].Revision);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void DeployArtifacts_NullSecrets_StaysNull()
|
||||
{
|
||||
var cmd = new DeployArtifactsCommand("dep", null, null, null, null, null, null, DateTimeOffset.UtcNow);
|
||||
var dto = SiteCommandDtoMapper.ToProto(cmd);
|
||||
Assert.Null(dto.Secrets);
|
||||
Assert.Null(((DeployArtifactsCommand)SiteCommandDtoMapper.FromProto(dto)).Secrets);
|
||||
}
|
||||
|
||||
[Theory, InlineData(null), InlineData(3)]
|
||||
public void ArtifactResponse_SecretsApplied_RoundTrips(int? applied)
|
||||
{
|
||||
var r = new ArtifactDeploymentResponse("dep", "site-a", true, null, DateTimeOffset.UtcNow, applied);
|
||||
Assert.Equal(applied, SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(r)).SecretsApplied);
|
||||
}
|
||||
```
|
||||
|
||||
(Adapt the `FromProto`/`ToProto` call shapes to the mapper's actual overloads for the lifecycle request and the response DTO — the existing `DeployArtifacts` mapping at `SiteCommandDtoMapper.cs:395` is the pattern.)
|
||||
|
||||
- [ ] **Step 4: Run → FAIL, implement mapper branches, run → PASS.** Byte fields via `ByteString.CopyFrom`/`.ToByteArray()`, timestamps via `Timestamp.FromDateTimeOffset`/`.ToDateTimeOffset()`, null list ⇒ leave field unset.
|
||||
|
||||
- [ ] **Step 5: Commit** — `git add src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Artifacts src/ZB.MOM.WW.ScadaBridge.Communication tests/ZB.MOM.WW.ScadaBridge.Communication.Tests && git commit -m "feat(artifacts): additive Secrets list + SecretsApplied on the artifact deploy contract (proto field 9)"`
|
||||
|
||||
```json:metadata
|
||||
{"files": ["src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Artifacts/SecretArtifact.cs", "src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Artifacts/DeployArtifactsCommand.cs", "src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Artifacts/ArtifactDeploymentResponse.cs", "src/ZB.MOM.WW.ScadaBridge.Communication/Protos/site_command.proto", "src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteCommandDtoMapper.cs"], "verifyCommand": "dotnet test tests/ZB.MOM.WW.ScadaBridge.Communication.Tests --filter FullyQualifiedName~SiteCommandDtoMapperSecretsTests", "acceptanceCriteria": ["SecretArtifact round-trips byte-for-byte", "null Secrets stays null / absent", "SecretsApplied null and 3 round-trip", "no active Protobuf item committed"], "modelTier": "standard"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 3: `SecretReferenceScanner`
|
||||
|
||||
**Goal:** A pure function returning, per site, the distinct `secret:<name>` references found anywhere in its data connections' primary/backup JSON, each with the referencing connection names.
|
||||
|
||||
**Files:**
|
||||
- Create: `src/ZB.MOM.WW.ScadaBridge.Commons/Types/Secrets/SecretReferenceScanner.cs`
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Types/Secrets/SecretReferenceScannerTests.cs`
|
||||
|
||||
**Acceptance Criteria:**
|
||||
- [ ] Finds `secret:` string values at any depth (objects and arrays), prefix case-insensitive, name trimmed.
|
||||
- [ ] Ignores non-string values, strings merely containing `secret:` mid-text, `${secret:X}` (node-local config form), empty names, and unparseable JSON (no throw).
|
||||
- [ ] Output ordinal-sorted by name; each entry lists referencing connections ordinal-sorted and de-duplicated.
|
||||
- [ ] Bootstrap names (`SB-GRPC-PSK-` prefix) are excluded.
|
||||
|
||||
**Verify:** `dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests --filter FullyQualifiedName~SecretReferenceScannerTests` → all pass.
|
||||
|
||||
**Steps:**
|
||||
|
||||
- [ ] **Step 1: Failing tests.**
|
||||
|
||||
```csharp
|
||||
private static DataConnectionArtifact Dc(string name, string? primary, string? backup = null)
|
||||
=> new(name, "MxGateway", primary, backup, 3);
|
||||
|
||||
[Fact]
|
||||
public void FindsNestedRefs_CaseInsensitive_AcrossPrimaryAndBackup()
|
||||
{
|
||||
var refs = SecretReferenceScanner.Scan([
|
||||
Dc("GW1", """{"Endpoint":"http://gw:5000","ApiKey":"SECRET: mxgw-key "}"""),
|
||||
Dc("GW2", """{"a":{"b":["x","secret:mxgw-key"]}}""", """{"ApiKey":"secret:backup-key"}"""),
|
||||
]);
|
||||
Assert.Equal(["backup-key", "mxgw-key"], refs.Select(r => r.Name));
|
||||
Assert.Equal(["GW1", "GW2"], refs.Single(r => r.Name == "mxgw-key").ReferencedBy);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData("""{"ApiKey":"${secret:node-local}"}""")]
|
||||
[InlineData("""{"Note":"see secret:x in docs"}""")]
|
||||
[InlineData("""{"ApiKey":"secret:"}""")]
|
||||
[InlineData("""{"Port":5000}""")]
|
||||
[InlineData("not json")]
|
||||
[InlineData("""{"Psk":"secret:SB-GRPC-PSK-site-a"}""")]
|
||||
public void IgnoresNonReferences(string json)
|
||||
=> Assert.Empty(SecretReferenceScanner.Scan([Dc("C", json)]));
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run → FAIL.**
|
||||
|
||||
- [ ] **Step 3: Implement.**
|
||||
|
||||
```csharp
|
||||
using System.Text.Json;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Commons.Types.Secrets;
|
||||
|
||||
/// <summary>One secret a site needs, and which of its data connections reference it.</summary>
|
||||
public sealed record SecretReference(string Name, IReadOnlyList<string> ReferencedBy);
|
||||
|
||||
/// <summary>
|
||||
/// Derives the secrets a site needs from its data-connection configs: every JSON string VALUE of the
|
||||
/// exact form <c>secret:<name></c>, at any depth, in primary or backup configuration. Protocol-agnostic
|
||||
/// so a new adapter adopting <c>secret:</c> refs needs no change here. Bootstrap secrets are excluded
|
||||
/// (design §3.1). Unparseable JSON yields nothing rather than throwing — the deploy gate owns config validity.
|
||||
/// </summary>
|
||||
public static class SecretReferenceScanner
|
||||
{
|
||||
public const string Prefix = "secret:";
|
||||
private static readonly string[] BootstrapPrefixes = ["SB-GRPC-PSK-"];
|
||||
|
||||
public static IReadOnlyList<SecretReference> Scan(IEnumerable<DataConnectionArtifact> connections)
|
||||
{
|
||||
var found = new SortedDictionary<string, SortedSet<string>>(StringComparer.Ordinal);
|
||||
foreach (var dc in connections)
|
||||
foreach (var json in new[] { dc.PrimaryConfigurationJson, dc.BackupConfigurationJson })
|
||||
foreach (var name in NamesIn(json))
|
||||
{
|
||||
if (!found.TryGetValue(name, out var by)) found[name] = by = new SortedSet<string>(StringComparer.Ordinal);
|
||||
by.Add(dc.Name);
|
||||
}
|
||||
return found.Select(kv => new SecretReference(kv.Key, kv.Value.ToList())).ToList();
|
||||
}
|
||||
|
||||
private static IEnumerable<string> NamesIn(string? json)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(json)) yield break;
|
||||
JsonDocument doc;
|
||||
try { doc = JsonDocument.Parse(json); } catch (JsonException) { yield break; }
|
||||
using (doc)
|
||||
{
|
||||
var stack = new Stack<JsonElement>(); stack.Push(doc.RootElement);
|
||||
while (stack.Count > 0)
|
||||
{
|
||||
var e = stack.Pop();
|
||||
switch (e.ValueKind)
|
||||
{
|
||||
case JsonValueKind.Object: foreach (var p in e.EnumerateObject()) stack.Push(p.Value); break;
|
||||
case JsonValueKind.Array: foreach (var i in e.EnumerateArray()) stack.Push(i); break;
|
||||
case JsonValueKind.String:
|
||||
var s = e.GetString()!.Trim();
|
||||
if (!s.StartsWith(Prefix, StringComparison.OrdinalIgnoreCase)) break;
|
||||
var name = s[Prefix.Length..].Trim();
|
||||
if (name.Length == 0 || BootstrapPrefixes.Any(b => name.StartsWith(b, StringComparison.OrdinalIgnoreCase))) break;
|
||||
yield return name;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
(`DataConnectionArtifact` property names: confirm `PrimaryConfigurationJson`/`BackupConfigurationJson` in `Commons/Messages/Artifacts/DataConnectionArtifact.cs` and adjust.)
|
||||
|
||||
- [ ] **Step 4: Run → PASS.** **Step 5: Commit** — `git add src/ZB.MOM.WW.ScadaBridge.Commons/Types/Secrets tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Types/Secrets && git commit -m "feat(secrets): SecretReferenceScanner derives a site's secret refs from its data-connection configs"`
|
||||
|
||||
```json:metadata
|
||||
{"files": ["src/ZB.MOM.WW.ScadaBridge.Commons/Types/Secrets/SecretReferenceScanner.cs", "tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Types/Secrets/SecretReferenceScannerTests.cs"], "verifyCommand": "dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests --filter FullyQualifiedName~SecretReferenceScannerTests", "acceptanceCriteria": ["nested/case-insensitive refs found in primary+backup", "non-refs, ${secret:}, bad JSON, bootstrap names ignored", "sorted, de-duplicated output"], "modelTier": "mechanical"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Secrets in the artifact revision hash
|
||||
|
||||
**Goal:** `ArtifactRevisionHashService` folds the command's secrets (and missing-secret markers) into the hash; sites with no secret refs keep their existing hash.
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.DeploymentManager/ArtifactRevisionHashService.cs`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Artifacts/DeployArtifactsCommand.cs` (append `IReadOnlyList<string>? MissingSecrets = null` — central-only, never mapped to proto; document that on the parameter)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests/ArtifactRevisionHashServiceTests.cs`
|
||||
|
||||
**Acceptance Criteria:**
|
||||
- [ ] A command with `Secrets == null` and `MissingSecrets == null` hashes identically to the pre-change golden value (add a golden-hash test computed on `main` BEFORE the change).
|
||||
- [ ] Changing a secret's revision, ciphertext, wrapped DEK or KekId changes the hash; reordering secrets does not.
|
||||
- [ ] A missing name hashes differently from the same name present.
|
||||
- [ ] `HashableRecords_PropertiesDeclaredAlphabetically` covers `HashableSecret` and passes.
|
||||
|
||||
**Verify:** `dotnet test tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests --filter FullyQualifiedName~ArtifactRevisionHashServiceTests` → all pass.
|
||||
|
||||
**Steps:**
|
||||
|
||||
- [ ] **Step 1: Golden test first (on unchanged code):** build a fixed command with one of each existing artifact kind, `Assert.Equal("<value printed on first run>", svc.ComputeHash(cmd))`; run once to capture the value, paste it in, commit the test alone (`test: pin pre-secrets artifact hash`).
|
||||
|
||||
- [ ] **Step 2: Failing tests.**
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public void SecretChanges_MoveTheHash_ButOrderDoesNot()
|
||||
{
|
||||
var svc = new ArtifactRevisionHashService();
|
||||
var a = Secret("A", rev: 1); var b = Secret("B", rev: 1);
|
||||
var baseline = svc.ComputeHash(Cmd(secrets: [a, b]));
|
||||
Assert.Equal(baseline, svc.ComputeHash(Cmd(secrets: [b, a])));
|
||||
Assert.NotEqual(baseline, svc.ComputeHash(Cmd(secrets: [a with { Revision = 2 }, b])));
|
||||
Assert.NotEqual(baseline, svc.ComputeHash(Cmd(secrets: [a with { WrappedDek = [42] }, b])));
|
||||
Assert.NotEqual(baseline, svc.ComputeHash(Cmd(secrets: [a with { KekId = "kek-B" }, b])));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void MissingSecret_HashesDifferentlyFromPresent()
|
||||
{
|
||||
var svc = new ArtifactRevisionHashService();
|
||||
Assert.NotEqual(svc.ComputeHash(Cmd(secrets: [Secret("A", 1)])), svc.ComputeHash(Cmd(missing: ["A"])));
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Implement.** Add to `HashableArtifactSet` (keep alphabetical: `DataConnections, DatabaseConnections, ExternalSystems, Secrets, SharedScripts`):
|
||||
|
||||
```csharp
|
||||
Secrets = BuildHashableSecrets(command),
|
||||
...
|
||||
private static List<HashableSecret>? BuildHashableSecrets(DeployArtifactsCommand c)
|
||||
{
|
||||
var list = new List<HashableSecret>();
|
||||
foreach (var s in c.Secrets ?? [])
|
||||
list.Add(new HashableSecret
|
||||
{
|
||||
ContentDigest = Convert.ToHexStringLower(SHA256.HashData([.. s.Ciphertext, .. s.Nonce, .. s.Tag, .. s.WrappedDek, .. s.WrapNonce, .. s.WrapTag])),
|
||||
KekId = s.KekId, Missing = null, Name = s.Name, Revision = s.Revision
|
||||
});
|
||||
foreach (var m in c.MissingSecrets ?? [])
|
||||
list.Add(new HashableSecret { Missing = true, Name = m });
|
||||
return list.Count == 0 ? null : list.OrderBy(x => x.Name, StringComparer.Ordinal).ThenBy(x => x.Missing == true).ToList();
|
||||
}
|
||||
|
||||
private sealed record HashableSecret
|
||||
{
|
||||
public string? ContentDigest { get; init; }
|
||||
public string? KekId { get; init; }
|
||||
public bool? Missing { get; init; }
|
||||
public required string Name { get; init; }
|
||||
public long? Revision { get; init; }
|
||||
}
|
||||
```
|
||||
|
||||
Update the class doc's EXCLUDED/INCLUDED paragraphs (secrets included as a ciphertext digest; `MissingSecrets` included as markers).
|
||||
|
||||
- [ ] **Step 4: Run → PASS (incl. golden).** **Step 5: Commit** — `git commit -m "feat(artifacts): secrets and missing-secret markers in the artifact revision hash" -- src/ZB.MOM.WW.ScadaBridge.DeploymentManager/ArtifactRevisionHashService.cs src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Artifacts/DeployArtifactsCommand.cs tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests/ArtifactRevisionHashServiceTests.cs`
|
||||
|
||||
```json:metadata
|
||||
{"files": ["src/ZB.MOM.WW.ScadaBridge.DeploymentManager/ArtifactRevisionHashService.cs", "tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests/ArtifactRevisionHashServiceTests.cs"], "verifyCommand": "dotnet test tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests --filter FullyQualifiedName~ArtifactRevisionHashServiceTests", "acceptanceCriteria": ["golden pre-change hash unchanged when no secrets", "revision/ciphertext/wrapped DEK/KekId changes move the hash; order does not", "missing marker differs from present", "alphabetical-properties test passes"], "modelTier": "standard"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 5: Central shared secret store — `Secrets:Store:Provider`
|
||||
|
||||
**Goal:** A Central node can use the shared SQL-Server secret store without the hub; a clustered Central on `Sqlite` warns.
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/SecretsRegistration.cs`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/StartupValidator.cs`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/appsettings.json` (`"Store": { "Provider": "Sqlite" }` under `Secrets`, with an `_comment`)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SecretsReplicationWiringTests.cs` (extend), `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/StartupValidatorTests.cs` (extend)
|
||||
|
||||
**Acceptance Criteria:**
|
||||
- [ ] Central + `Provider=SqlServer` + connection string → `ISecretStore` resolves to the library's SQL-Server store; no hub endpoint mapped.
|
||||
- [ ] Central + `Provider=SqlServer` + blank connection string → startup failure naming `Secrets:SqlServer:ConnectionString`; a `${secret:` value is rejected (bootstrap rule, same message style as the existing Grpc-mode check).
|
||||
- [ ] Site role ignores `Secrets:Store:Provider` (always local SQLite).
|
||||
- [ ] Clustered Central (seed-nodes > 1) with `Provider=Sqlite` logs one Warning naming the per-node divergence risk; unknown provider value fails validation.
|
||||
|
||||
**Verify:** `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter "FullyQualifiedName~SecretsReplicationWiringTests|FullyQualifiedName~StartupValidatorTests"` → all pass.
|
||||
|
||||
**Steps:**
|
||||
|
||||
- [ ] **Step 1: Failing tests** modelled on the existing Grpc-mode central tests in `SecretsReplicationWiringTests` (they already build a service collection with `AddScadaBridgeSecrets(config, SecretsNodeRole.Central)` and assert the resolved `ISecretStore` type) — add `Central_StoreProviderSqlServer_UsesSharedSqlStore_WithoutHub`, `Central_StoreProviderSqlServer_BlankConnectionString_Throws`, `Central_StoreProviderSqlServer_SecretReferenceConnectionString_Throws`, `Site_IgnoresStoreProvider`. In `StartupValidatorTests` add `ClusteredCentral_SqliteSecretStore_Warns` and `UnknownSecretStoreProvider_Fails`.
|
||||
|
||||
- [ ] **Step 2: Run → FAIL.**
|
||||
|
||||
- [ ] **Step 3: Implement.** In `SecretsRegistration`: add `public const string StoreProviderKey = "Secrets:Store:Provider";` and, in `AddScadaBridgeSecrets`, BEFORE the replication branch: when `role == SecretsNodeRole.Central` and the provider equals `SqlServer` (ordinal-ignore-case) and replication is not enabled, validate the connection string with the same helper the Grpc-central branch uses (blank / `${secret:` rejection), then call `services.AddZbSecretsSqlServerStore(...)` with the same arguments the Grpc-central branch passes, then `services.AddZbSecrets(...)` exactly as that branch does, and return. Keep the Grpc branch untouched here (Task 11 retires it). In `StartupValidator`: validate the provider is `Sqlite|SqlServer`; for a Central with > 1 seed node and `Sqlite`, log the Warning `Secrets use a per-node SQLite store on a clustered central — each node would deploy its own secret set to sites; set Secrets:Store:Provider=SqlServer`.
|
||||
|
||||
- [ ] **Step 4: Run → PASS.** **Step 5: Commit** — `git commit -m "feat(secrets): Secrets:Store:Provider=SqlServer — central pair shares one SQL secret store without the hub" -- src/ZB.MOM.WW.ScadaBridge.Host tests/ZB.MOM.WW.ScadaBridge.Host.Tests`
|
||||
|
||||
```json:metadata
|
||||
{"files": ["src/ZB.MOM.WW.ScadaBridge.Host/SecretsRegistration.cs", "src/ZB.MOM.WW.ScadaBridge.Host/StartupValidator.cs", "src/ZB.MOM.WW.ScadaBridge.Host/appsettings.json"], "verifyCommand": "dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter \"FullyQualifiedName~SecretsReplicationWiringTests|FullyQualifiedName~StartupValidatorTests\"", "acceptanceCriteria": ["Central+SqlServer resolves shared SQL store, no hub", "blank or ${secret:} connection string fails naming the key", "Site ignores provider", "clustered Central on Sqlite warns; unknown provider fails"], "modelTier": "standard"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 6: Central — collect a site's secrets, fail closed, log
|
||||
|
||||
**Goal:** `ArtifactDeploymentService` adds each site's referenced secrets (ciphertext rows from `ISecretStore`) to its command, fails that site closed on a missing secret, logs names/revisions, and `ComputeCurrentArtifactHashesAsync` includes missing markers.
|
||||
|
||||
**Files:**
|
||||
- Create: `src/ZB.MOM.WW.ScadaBridge.DeploymentManager/SiteSecretsCollector.cs`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.DeploymentManager/ArtifactDeploymentService.cs` (`BuildDeployArtifactsCommandAsync`, `DeployCoreAsync`, `RetryForSiteAsync`, `ComputeCurrentArtifactHashesAsync`)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.DeploymentManager/ZB.MOM.WW.ScadaBridge.DeploymentManager.csproj` (`<PackageReference Include="ZB.MOM.WW.Secrets.Abstractions" />`), DeploymentManager DI registration (`SiteSecretsCollector` singleton)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests/SiteSecretsCollectorTests.cs` (new), `ArtifactDeploymentServiceTests.cs` (extend)
|
||||
|
||||
**Acceptance Criteria:**
|
||||
- [ ] A site whose data connections reference `A` and `B` gets a command whose `Secrets` are exactly A and B's `StoredSecret` rows mapped field-for-field; other central secrets are not included.
|
||||
- [ ] A referenced name that is absent or `IsDeleted` → that site's `SiteArtifactResult` fails with `Secret 'X' referenced by data connection(s) C1, C2 is not in the central secret store`, nothing is sent to that site, other sites in the batch still deploy.
|
||||
- [ ] `ComputeCurrentArtifactHashesAsync` puts missing names in `MissingSecrets` instead of failing, so such a site reads *Out of date*.
|
||||
- [ ] Central logs Information `Deploying {Count} secret(s) to site {SiteId}: {Secrets}` (formatted `name@rN`) and Error for missing; a test asserts no log/exception text contains the secret's plaintext or base64 ciphertext.
|
||||
|
||||
**Verify:** `dotnet test tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests --filter "FullyQualifiedName~SiteSecretsCollectorTests|FullyQualifiedName~ArtifactDeploymentServiceTests"` → all pass.
|
||||
|
||||
**Steps:**
|
||||
|
||||
- [ ] **Step 1: Failing tests** for the collector with an in-memory fake `ISecretStore` (implement only `GetAsync`; other members throw `NotSupportedException`):
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task Collect_ReturnsOnlyReferencedRows_AndNamesMissing()
|
||||
{
|
||||
var store = new FakeSecretStore(Row("A", 3), Row("B", 1, deleted: true), Row("OTHER", 1));
|
||||
var result = await new SiteSecretsCollector(store).CollectAsync(
|
||||
[Dc("GW1", """{"ApiKey":"secret:A"}"""), Dc("GW2", """{"ApiKey":"secret:B"}""")], default);
|
||||
Assert.Equal(["A"], result.Secrets.Select(s => s.Name));
|
||||
Assert.Equal(3, result.Secrets[0].Revision);
|
||||
Assert.Equal(["B"], result.Missing.Select(m => m.Name));
|
||||
Assert.Equal(["GW2"], result.Missing[0].ReferencedBy);
|
||||
}
|
||||
```
|
||||
|
||||
And in `ArtifactDeploymentServiceTests`: `DeployToAllSites_SiteWithMissingSecret_FailsThatSiteOnly_AndSendsNothingToIt` (assert the communication fake received no command for that site and the other site succeeded), `ComputeCurrentHashes_MissingSecret_UsesMissingMarker` (hash differs from the one computed once the secret exists), `DeployLogs_NeverContainSecretMaterial`.
|
||||
|
||||
- [ ] **Step 2: Run → FAIL.**
|
||||
|
||||
- [ ] **Step 3: Implement the collector.**
|
||||
|
||||
```csharp
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Secrets;
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.DeploymentManager;
|
||||
|
||||
public sealed record SiteSecretsCollection(IReadOnlyList<SecretArtifact> Secrets, IReadOnlyList<SecretReference> Missing);
|
||||
|
||||
/// <summary>Central half of design §3.1/§3.3: which secrets a site needs and their ciphertext rows. Never decrypts.</summary>
|
||||
public sealed class SiteSecretsCollector(ISecretStore store)
|
||||
{
|
||||
public async Task<SiteSecretsCollection> CollectAsync(IEnumerable<DataConnectionArtifact> connections, CancellationToken ct)
|
||||
{
|
||||
var secrets = new List<SecretArtifact>(); var missing = new List<SecretReference>();
|
||||
foreach (var reference in SecretReferenceScanner.Scan(connections))
|
||||
{
|
||||
var row = await store.GetAsync(new SecretName(reference.Name), ct);
|
||||
if (row is null || row.IsDeleted) { missing.Add(reference); continue; }
|
||||
secrets.Add(new SecretArtifact(row.Name.Value, row.ContentType.ToString(), row.Ciphertext, row.Nonce, row.Tag,
|
||||
row.WrappedDek, row.WrapNonce, row.WrapTag, row.KekId, row.Revision, row.CreatedUtc, row.UpdatedUtc, row.Description));
|
||||
}
|
||||
return new SiteSecretsCollection(secrets, missing);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
(`SecretName` is a `readonly record struct` that normalizes its value at construction — always ship `row.Name.Value`, the normalized form. `SecretContentType` is an enum; ship `ToString()`, parsed back in Task 9.)
|
||||
|
||||
In `BuildDeployArtifactsCommandAsync`, after mapping data connections: `var collected = await _secretsCollector.CollectAsync(dataConnectionArtifacts, ct);` and pass `Secrets: collected.Secrets.Count > 0 ? collected.Secrets : null, MissingSecrets: collected.Missing.Count > 0 ? collected.Missing.Select(m => m.Name).ToList() : null`. Return the missing list alongside the command (change the private method to return `(DeployArtifactsCommand Command, IReadOnlyList<SecretReference> Missing)`). In `DeployCoreAsync` and `RetryForSiteAsync`: if `Missing.Count > 0`, log Error and produce a failed `SiteArtifactResult` with the message from the acceptance criteria WITHOUT calling `_communicationService`; else log the Information line (only when secrets were present) and send. `ComputeCurrentArtifactHashesAsync` keeps using the command (missing markers included).
|
||||
|
||||
- [ ] **Step 4: Run → PASS.** **Step 5: Commit** — `git commit -m "feat(artifacts): central ships each site's referenced secrets (ciphertext) and fails closed on a missing one" -- src/ZB.MOM.WW.ScadaBridge.DeploymentManager tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests`
|
||||
|
||||
```json:metadata
|
||||
{"files": ["src/ZB.MOM.WW.ScadaBridge.DeploymentManager/SiteSecretsCollector.cs", "src/ZB.MOM.WW.ScadaBridge.DeploymentManager/ArtifactDeploymentService.cs"], "verifyCommand": "dotnet test tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests --filter \"FullyQualifiedName~SiteSecretsCollectorTests|FullyQualifiedName~ArtifactDeploymentServiceTests\"", "acceptanceCriteria": ["only referenced rows shipped, field-for-field", "missing/tombstoned ref fails only that site with the named message, nothing sent", "current-hash uses missing markers", "logs name@revision, never material"], "modelTier": "standard"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 7: Central — stamp gating for old sites + audit row
|
||||
|
||||
**Goal:** Central never stamps a site *Current* unless the site confirmed applying every shipped secret, and every secret deployment writes one audit row.
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.DeploymentManager/ArtifactDeploymentService.cs`
|
||||
- Modify: `SiteArtifactResult` record (same file or its Commons location) — append `int? SecretsDeployed = null`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.CLI/Commands/DeployCommands.cs` only if its table output enumerates columns (JSON output carries the new field automatically)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests/ArtifactDeploymentServiceTests.cs`
|
||||
|
||||
**Acceptance Criteria:**
|
||||
- [ ] Command with ≥ 1 secret + response `Success=true, SecretsApplied=null` → site result is a failure `Site build predates secret deployment — upgrade the site before deploying secrets`; hash NOT stamped.
|
||||
- [ ] `SecretsApplied` ≠ shipped count → failure `Site applied {n} of {m} secrets`; not stamped.
|
||||
- [ ] Command with no secrets → old-site response (`SecretsApplied=null`) still succeeds and stamps (unchanged behaviour).
|
||||
- [ ] One `IAuditService.LogAsync(user, "DeploySiteSecrets", "SiteSecrets", siteIdentifier, siteName, { deploymentId, secrets:[{name,revision}], missing:[name], success, error })` per site whose command carried or would have carried secrets (incl. the fail-closed case); none for sites with no refs. Audit has no material.
|
||||
|
||||
**Verify:** `dotnet test tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests --filter FullyQualifiedName~ArtifactDeploymentServiceTests` → all pass.
|
||||
|
||||
**Steps:**
|
||||
|
||||
- [ ] **Step 1: Failing tests** — `OldSite_NullSecretsApplied_WithSecrets_FailsAndDoesNotStamp`, `PartialApply_FailsAndDoesNotStamp`, `OldSite_NoSecrets_StillStamps`, `SecretDeploy_WritesOneAuditRowPerSite_WithoutMaterial` (capture `IAuditService` calls in the existing test fake; serialize `afterState` and assert it contains the names/revisions and not the base64 ciphertext).
|
||||
|
||||
- [ ] **Step 2: Run → FAIL.**
|
||||
|
||||
- [ ] **Step 3: Implement** a private `static SiteArtifactResult ReconcileSecretsOutcome(DeployArtifactsCommand cmd, ArtifactDeploymentResponse r, Site site)` applied right after each `DeployArtifactsAsync` response in both paths:
|
||||
|
||||
```csharp
|
||||
var shipped = cmd.Secrets?.Count ?? 0;
|
||||
if (r.Success && shipped > 0 && r.SecretsApplied is null)
|
||||
return new(site.SiteIdentifier, site.Name, false, "Site build predates secret deployment — upgrade the site before deploying secrets", null);
|
||||
if (r.Success && shipped > 0 && r.SecretsApplied != shipped)
|
||||
return new(site.SiteIdentifier, site.Name, false, $"Site applied {r.SecretsApplied} of {shipped} secrets", r.SecretsApplied);
|
||||
return new(site.SiteIdentifier, site.Name, r.Success, r.ErrorMessage, shipped > 0 ? r.SecretsApplied : null);
|
||||
```
|
||||
|
||||
Stamping already keys off `Success`, so a reconciled failure is not stamped. Add the audit call after the per-site results are known (after the existing `DeployArtifacts` audit row, same ordering rule: audit first, best-effort stamp after).
|
||||
|
||||
- [ ] **Step 4: Run → PASS.** **Step 5: Commit** — `git commit -m "feat(artifacts): never stamp a site current unless it applied every secret; DeploySiteSecrets audit row" -- src/ZB.MOM.WW.ScadaBridge.DeploymentManager src/ZB.MOM.WW.ScadaBridge.CLI tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests`
|
||||
|
||||
```json:metadata
|
||||
{"files": ["src/ZB.MOM.WW.ScadaBridge.DeploymentManager/ArtifactDeploymentService.cs"], "verifyCommand": "dotnet test tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests --filter FullyQualifiedName~ArtifactDeploymentServiceTests", "acceptanceCriteria": ["old site + secrets -> failure, not stamped", "partial apply -> failure, not stamped", "no secrets -> unchanged behaviour", "one DeploySiteSecrets audit row per secret-bearing site, no material"], "modelTier": "standard"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 8: Site storage — `deployed_secrets` (replicated) + `mirrored_secrets` (local)
|
||||
|
||||
**Goal:** The site database has a replicated ciphertext table for deployed secrets and a node-local ledger of mirrored names, with full-set accessors.
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageSchema.cs`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs` (`"deployed_secrets"` appended to `ReplicatedTables`; update the "ten tables" comments to eleven; `mirrored_secrets` deliberately NOT registered, with a comment why)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/DeployedSecretsStorageTests.cs` (new); existing `SiteLocalDbReplicatedTableFkGuardTests` must still pass
|
||||
|
||||
**Acceptance Criteria:**
|
||||
- [ ] `UpsertDeployedSecretsAsync(IReadOnlyList<SecretArtifact>, deploymentId)` upserts rows (`is_removed = 0`) and marks every other row `is_removed = 1`; returns `(applied names, newly removed names)`.
|
||||
- [ ] `GetDeployedSecretsAsync()` returns all rows incl. removed with their `SecretArtifact` round-tripped from `row_json`.
|
||||
- [ ] `mirrored_secrets` get/add/remove work; it is not in `ReplicatedTables`.
|
||||
- [ ] A security test asserts `row_json` holds only base64 ciphertext fields (deserializes to `SecretArtifact`, no extra properties).
|
||||
- [ ] FK guard test green (no FKs on either table).
|
||||
|
||||
**Verify:** `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter FullyQualifiedName~DeployedSecretsStorageTests` and `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter FullyQualifiedName~SiteLocalDbReplicatedTableFkGuardTests` → all pass.
|
||||
|
||||
**Steps:**
|
||||
|
||||
- [ ] **Step 1: Schema** (append to the `CREATE TABLE IF NOT EXISTS` script):
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS deployed_secrets (
|
||||
name TEXT PRIMARY KEY,
|
||||
row_json TEXT NOT NULL,
|
||||
revision INTEGER NOT NULL,
|
||||
updated_utc TEXT NOT NULL,
|
||||
is_removed INTEGER NOT NULL DEFAULT 0,
|
||||
deployment_id TEXT NOT NULL,
|
||||
applied_at TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS mirrored_secrets (
|
||||
name TEXT PRIMARY KEY,
|
||||
revision INTEGER NOT NULL,
|
||||
mirrored_at TEXT NOT NULL
|
||||
);
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Failing tests** against a temp-file `SiteStorageService` (follow the existing SiteStorageService test fixture in `SiteRuntime.Tests/Persistence`):
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task Upsert_IsFullSet_MarksAbsentRowsRemoved()
|
||||
{
|
||||
var s = await Fixture.CreateAsync();
|
||||
await s.UpsertDeployedSecretsAsync([Art("A", 1), Art("B", 1)], "d1");
|
||||
var (applied, removed) = await s.UpsertDeployedSecretsAsync([Art("A", 2)], "d2");
|
||||
Assert.Equal(["A"], applied); Assert.Equal(["B"], removed);
|
||||
var rows = await s.GetDeployedSecretsAsync();
|
||||
Assert.Equal(2, rows.Single(r => r.Name == "A").Secret.Revision);
|
||||
Assert.True(rows.Single(r => r.Name == "B").IsRemoved);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Implement** `DeployedSecretRow(string Name, SecretArtifact Secret, bool IsRemoved, string DeploymentId)` and the accessors (single transaction; `row_json` via `JsonSerializer.Serialize(secretArtifact)` — byte[] serializes as base64). Add `GetMirroredSecretsAsync`, `SetMirroredSecretAsync(name, revision)`, `RemoveMirroredSecretAsync(name)`. Register `deployed_secrets` in `ReplicatedTables`.
|
||||
|
||||
- [ ] **Step 4: Run → PASS.** **Step 5: Commit** — `git commit -m "feat(site): deployed_secrets (LocalDb-replicated ciphertext) + node-local mirrored_secrets ledger" -- src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence`
|
||||
|
||||
```json:metadata
|
||||
{"files": ["src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageSchema.cs", "src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs", "src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs"], "verifyCommand": "dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter FullyQualifiedName~DeployedSecretsStorageTests", "acceptanceCriteria": ["full-set upsert marks absent rows removed", "rows round-trip SecretArtifact", "mirrored_secrets local only", "row_json ciphertext-only security test", "FK guard green"], "modelTier": "standard"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 9: `DeployedSecretsMirror` — per-node apply + verify
|
||||
|
||||
**Goal:** Every site node reconciles `deployed_secrets` into its own `secrets.db` (apply live rows, delete removed rows it previously mirrored), verifies each deployed secret decrypts, logs changes, and can be invoked synchronously.
|
||||
|
||||
**Files:**
|
||||
- Create: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Secrets/DeployedSecretsMirror.cs` (service + `IHostedService` loop)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs` (+ `DeployedSecretsMirrorInterval`, default 15 s) and `SiteRuntimeOptionsValidator.cs` (> 0)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ZB.MOM.WW.ScadaBridge.SiteRuntime.csproj` (`ZB.MOM.WW.Secrets.Abstractions`), SiteRuntime DI (`AddSingleton<DeployedSecretsMirror>()` + `AddHostedService(sp => sp.GetRequiredService<DeployedSecretsMirror>())`)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Secrets/DeployedSecretsMirrorTests.cs`
|
||||
|
||||
**Acceptance Criteria:**
|
||||
- [ ] `ReconcileAsync()` applies each live row with `ISecretStore.ApplyReplicatedAsync` (mapped back to `StoredSecret`), records it in `mirrored_secrets`, and returns a `MirrorResult(Applied, Removed, Failed)`.
|
||||
- [ ] A removed row is deleted from the store only if its name is in `mirrored_secrets`; a same-named out-of-band secret never mirrored is left alone.
|
||||
- [ ] Each live secret is trial-resolved via `ISecretResolver`; a failure (null or exception) is a `Failed` entry `Secret '{Name}' (kek {KekId}) did not decrypt on this node — the node's master key differs from central's` and is logged at Error.
|
||||
- [ ] Logging: Information per applied/removed secret and one summary when anything changed; Debug when in sync; never values.
|
||||
- [ ] The hosted loop runs once at start and every `DeployedSecretsMirrorInterval`; a reconcile exception is logged and the loop continues.
|
||||
|
||||
**Verify:** `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter FullyQualifiedName~DeployedSecretsMirrorTests` → all pass.
|
||||
|
||||
**Steps:**
|
||||
|
||||
- [ ] **Step 1: Failing tests** using a real temp `SiteStorageService`, an in-memory fake `ISecretStore` recording `ApplyReplicatedAsync`/`DeleteAsync`, and a fake `ISecretResolver`:
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task Reconcile_Applies_Verifies_AndOnlyDeletesWhatItMirrored()
|
||||
{
|
||||
var (mirror, storage, store, resolver) = Harness.Create();
|
||||
store.Seed("OUT-OF-BAND");
|
||||
await storage.UpsertDeployedSecretsAsync([Art("A", 1), Art("OUT-OF-BAND", 1)], "d1");
|
||||
var r1 = await mirror.ReconcileAsync(default);
|
||||
Assert.Equal(["A", "OUT-OF-BAND"], r1.Applied.OrderBy(x => x));
|
||||
await storage.UpsertDeployedSecretsAsync([], "d2");
|
||||
await storage.RemoveMirroredSecretAsync("OUT-OF-BAND"); // simulate: never mirrored by us
|
||||
var r2 = await mirror.ReconcileAsync(default);
|
||||
Assert.Equal(["A"], r2.Removed);
|
||||
Assert.True(store.Contains("OUT-OF-BAND"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Reconcile_DecryptFailure_IsReportedWithKekId()
|
||||
{
|
||||
var (mirror, storage, _, resolver) = Harness.Create();
|
||||
resolver.FailFor("A");
|
||||
await storage.UpsertDeployedSecretsAsync([Art("A", 1, kek: "kek-central")], "d1");
|
||||
var r = await mirror.ReconcileAsync(default);
|
||||
Assert.Contains("kek-central", Assert.Single(r.Failed));
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run → FAIL.**
|
||||
|
||||
- [ ] **Step 3: Implement.**
|
||||
|
||||
```csharp
|
||||
public sealed record MirrorResult(IReadOnlyList<string> Applied, IReadOnlyList<string> Removed, IReadOnlyList<string> Failed)
|
||||
{
|
||||
public bool Changed => Applied.Count > 0 || Removed.Count > 0;
|
||||
}
|
||||
|
||||
/// <summary>Design §3.5 step 2: per-node reconcile of the replicated deployed_secrets table into this node's secret store.</summary>
|
||||
public sealed class DeployedSecretsMirror(
|
||||
SiteStorageService storage, ISecretStore store, ISecretResolver resolver,
|
||||
IOptions<SiteRuntimeOptions> options, ILogger<DeployedSecretsMirror> logger, TimeProvider time) : BackgroundService
|
||||
{
|
||||
private readonly SemaphoreSlim _gate = new(1, 1);
|
||||
|
||||
public async Task<MirrorResult> ReconcileAsync(CancellationToken ct)
|
||||
{
|
||||
await _gate.WaitAsync(ct);
|
||||
try
|
||||
{
|
||||
var rows = await storage.GetDeployedSecretsAsync();
|
||||
var mirrored = (await storage.GetMirroredSecretsAsync()).ToDictionary(m => m.Name, m => m.Revision, StringComparer.Ordinal);
|
||||
List<string> applied = [], removed = [], failed = [];
|
||||
foreach (var row in rows)
|
||||
{
|
||||
if (row.IsRemoved)
|
||||
{
|
||||
if (!mirrored.ContainsKey(row.Name)) continue;
|
||||
await store.DeleteAsync(new SecretName(row.Name), "artifact-deploy", ct);
|
||||
await storage.RemoveMirroredSecretAsync(row.Name);
|
||||
removed.Add(row.Name);
|
||||
logger.LogInformation("Removed deployed secret {SecretName} (no longer referenced by this site)", row.Name);
|
||||
continue;
|
||||
}
|
||||
if (!mirrored.TryGetValue(row.Name, out var rev) || rev != row.Secret.Revision)
|
||||
{
|
||||
await store.ApplyReplicatedAsync(ToStored(row.Secret), ct);
|
||||
await storage.SetMirroredSecretAsync(row.Name, row.Secret.Revision);
|
||||
applied.Add(row.Name);
|
||||
logger.LogInformation("Applied deployed secret {SecretName} r{Revision} (kek {KekId})", row.Name, row.Secret.Revision, row.Secret.KekId);
|
||||
}
|
||||
if (!await Resolves(row.Name, ct))
|
||||
{
|
||||
var msg = $"Secret '{row.Name}' (kek {row.Secret.KekId}) did not decrypt on this node — the node's master key differs from central's";
|
||||
failed.Add(msg);
|
||||
logger.LogError("{Failure}", msg);
|
||||
}
|
||||
}
|
||||
var result = new MirrorResult(applied, removed, failed);
|
||||
if (result.Changed) logger.LogInformation("Deployed secrets mirror: {Applied} applied, {Removed} removed, {Failed} failed", applied.Count, removed.Count, failed.Count);
|
||||
else logger.LogDebug("Deployed secrets mirror in sync ({Count} secret(s))", rows.Count(r => !r.IsRemoved));
|
||||
return result;
|
||||
}
|
||||
finally { _gate.Release(); }
|
||||
}
|
||||
|
||||
private async Task<bool> Resolves(string name, CancellationToken ct)
|
||||
{
|
||||
try { return await resolver.GetAsync(new SecretName(name), ct) is not null; }
|
||||
catch (Exception) { return false; }
|
||||
}
|
||||
|
||||
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
|
||||
{
|
||||
using var timer = new PeriodicTimer(options.Value.DeployedSecretsMirrorInterval, time);
|
||||
do
|
||||
{
|
||||
try { await ReconcileAsync(stoppingToken); }
|
||||
catch (Exception ex) when (ex is not OperationCanceledException) { logger.LogWarning(ex, "Deployed secrets mirror pass failed; retrying next interval"); }
|
||||
} while (await timer.WaitForNextTickAsync(stoppingToken));
|
||||
}
|
||||
|
||||
private static StoredSecret ToStored(SecretArtifact s) => new()
|
||||
{
|
||||
Name = new SecretName(s.Name), Description = s.Description, ContentType = Enum.Parse<SecretContentType>(s.ContentType, ignoreCase: false),
|
||||
Ciphertext = s.Ciphertext, Nonce = s.Nonce, Tag = s.Tag, WrappedDek = s.WrappedDek, WrapNonce = s.WrapNonce,
|
||||
WrapTag = s.WrapTag, KekId = s.KekId, Revision = s.Revision, CreatedUtc = s.CreatedUtc, UpdatedUtc = s.UpdatedUtc
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
`ContentType` is the `SecretContentType` enum (`Text|ConnectionString|Json|BinaryBase64`): Task 6 writes `row.ContentType.ToString()`, this method parses it back with `Enum.Parse<SecretContentType>` — add a round-trip unit test over all four values. Note: the resolver caches (`ResolveCacheTtl` 30 s); if a trial-resolve after a rotation returns the cached old value that is still a successful decrypt of *a* value — acceptable for the verify's purpose (KEK check), document it in the class remarks.
|
||||
|
||||
- [ ] **Step 4: Run → PASS.** **Step 5: Commit** — `git commit -m "feat(site): DeployedSecretsMirror — per-node apply + decrypt-verify of deployed secrets" -- src/ZB.MOM.WW.ScadaBridge.SiteRuntime tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Secrets`
|
||||
|
||||
```json:metadata
|
||||
{"files": ["src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Secrets/DeployedSecretsMirror.cs", "src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs", "src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptionsValidator.cs"], "verifyCommand": "dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter FullyQualifiedName~DeployedSecretsMirrorTests", "acceptanceCriteria": ["live rows applied via ApplyReplicatedAsync and ledgered", "removed rows deleted only if previously mirrored", "decrypt failure reported with KekId at Error", "change/in-sync logging without values", "hosted loop at start + interval, survives exceptions"], "modelTier": "standard"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 10: Site apply — secrets in `HandleDeployArtifacts`
|
||||
|
||||
**Goal:** The active site node writes shipped secrets to `deployed_secrets`, runs the mirror synchronously, and answers with `SecretsApplied` or a failure.
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs` (`HandleDeployArtifacts`, constructor gets `DeployedSecretsMirror?`)
|
||||
- Modify: the actor's Props/DI construction site to pass the mirror
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorArtifactSecretsTests.cs` (new; follow the existing artifact-apply actor tests)
|
||||
|
||||
**Acceptance Criteria:**
|
||||
- [ ] `Secrets == null` → `deployed_secrets` untouched, response `SecretsApplied = null` (old-central compatible).
|
||||
- [ ] `Secrets` present (incl. empty) → full-set write, synchronous `ReconcileAsync`, response `Success=true, SecretsApplied = shipped count` when the mirror reports no failures.
|
||||
- [ ] Any mirror failure → `Success=false`, `ErrorMessage` = the failure lines joined with `; `, `SecretsApplied` = count applied without failure.
|
||||
- [ ] Site Information log `Applying {Count} deployed secret(s), deploymentId={DeploymentId}` + the mirror's per-secret lines.
|
||||
|
||||
**Verify:** `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter FullyQualifiedName~DeploymentManagerActorArtifactSecretsTests` → all pass.
|
||||
|
||||
**Steps:**
|
||||
|
||||
- [ ] **Step 1: Failing tests** (TestKit; send `DeployArtifactsCommand` with 2 secrets, expect `ArtifactDeploymentResponse` with `SecretsApplied == 2`; a resolver fake failing one → `Success == false` and the KekId in the message; `Secrets == null` → `SecretsApplied == null` and table empty).
|
||||
|
||||
- [ ] **Step 2: Run → FAIL.**
|
||||
|
||||
- [ ] **Step 3: Implement** inside the existing `Task.Run` block, AFTER data connections are stored (so the DCL can resolve by the time it reconnects) and BEFORE the success reply:
|
||||
|
||||
```csharp
|
||||
int? secretsApplied = null;
|
||||
string? secretError = null;
|
||||
if (command.Secrets != null && _secretsMirror != null)
|
||||
{
|
||||
_logger.LogInformation("Applying {Count} deployed secret(s), deploymentId={DeploymentId}", command.Secrets.Count, command.DeploymentId);
|
||||
await _storage.UpsertDeployedSecretsAsync(command.Secrets, command.DeploymentId);
|
||||
var mirror = await _secretsMirror.ReconcileAsync(CancellationToken.None);
|
||||
var failedNames = mirror.Failed.Count;
|
||||
secretsApplied = command.Secrets.Count - failedNames;
|
||||
if (failedNames > 0) secretError = string.Join("; ", mirror.Failed);
|
||||
}
|
||||
```
|
||||
|
||||
and build the reply with `Success = secretError is null && <existing success>`, `ErrorMessage = secretError ?? <existing>`, `SecretsApplied = secretsApplied`. If the mirror is not registered (null) but secrets were shipped, reply failure `Secret deployment is not enabled on this site node`.
|
||||
|
||||
- [ ] **Step 4: Run → PASS**, plus the existing artifact-apply tests. **Step 5: Commit** — `git commit -m "feat(site): artifact apply writes deployed secrets and answers SecretsApplied" -- src/ZB.MOM.WW.ScadaBridge.SiteRuntime tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors`
|
||||
|
||||
```json:metadata
|
||||
{"files": ["src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs"], "verifyCommand": "dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter FullyQualifiedName~DeploymentManagerActorArtifactSecretsTests", "acceptanceCriteria": ["null Secrets leaves table and SecretsApplied null", "secrets -> full-set write + sync mirror + SecretsApplied=count", "mirror failure -> Success=false with KekId message", "apply logging"], "modelTier": "standard"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 11: Retire the gRPC hub for ScadaBridge; switch the rig
|
||||
|
||||
**Goal:** ScadaBridge no longer composes the secrets hub (client or server); config asking for it fails fast; the docker rig's central pair uses `Secrets:Store:Provider=SqlServer` and site-a gets secrets only via artifact deploy.
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/SecretsRegistration.cs` (delete the Grpc hub/hub-client and SqlServer-replication branches; keep local SQLite + Task 5's shared store), `Program.cs` (remove `MapScadaBridgeSecretsHub`), `StartupValidator.cs` (reject `Secrets:Replication:Enabled=true`: `Secrets replication (the gRPC hub) is retired in ScadaBridge — sites receive secrets with Deploy Artifacts; central uses Secrets:Store:Provider=SqlServer`)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/appsettings.json` (remove `Replication`/`GrpcHub` blocks; keep `SqlServer.ConnectionString`/`SchemaName`; update comments)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/ZB.MOM.WW.ScadaBridge.Host.csproj` (drop `ZB.MOM.WW.Secrets.Replicator.Grpc` reference) and `Directory.Packages.props` if nothing else uses it
|
||||
- Modify: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SecretsReplicationWiringTests.cs`, `MappedGrpcAuthenticationCoverageTests.cs` (hub route gone)
|
||||
- Modify: `docker/docker-compose.yml` (delete `x-secrets-hub-env` + `x-secrets-hub-site-env` anchors and their merges; central-a/b get `Secrets__Store__Provider: "SqlServer"` + the existing `x-secrets-hub-central-env` connection string renamed `x-secrets-central-store-env`; the rig KEK `ZB_SECRETS_MASTER_KEY` goes on central-a/b AND site-a-a/b — every node that handles deployed secrets must share it; site-b/site-c keep a DIFFERENT dev KEK so the Task 13 mismatch negative is real)
|
||||
|
||||
**Acceptance Criteria:**
|
||||
- [ ] No ScadaBridge code references `AddZbSecretsGrpcHub`, `AddZbSecretsGrpcHubClient` or `MapScadaBridgeSecretsHub`.
|
||||
- [ ] `Secrets:Replication:Enabled=true` on either role → startup failure with the message above.
|
||||
- [ ] Wiring tests cover: Central Sqlite, Central SqlServer store, Site local — all green; hub-route auth coverage test updated.
|
||||
- [ ] `docker compose config` renders; central-a/b carry `Secrets__Store__Provider=SqlServer`; no node carries `Secrets__Replication__*` or `Secrets__GrpcHub__*`.
|
||||
|
||||
**Verify:** `dotnet build ZB.MOM.WW.ScadaBridge.slnx` → 0 errors; `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests` → all pass; `docker compose -f docker/docker-compose.yml config | grep -c Secrets__Replication` → `0`.
|
||||
|
||||
**Steps:**
|
||||
|
||||
- [ ] **Step 1:** Update tests first (replace hub-mode wiring tests with `ReplicationEnabled_IsRejected_BothRoles`), run → FAIL.
|
||||
- [ ] **Step 2:** Remove the hub branches/mapping/package ref, add the validator rule, run → PASS.
|
||||
- [ ] **Step 3:** Edit `docker-compose.yml` as listed; `docker compose -f docker/docker-compose.yml config > /dev/null` → exit 0.
|
||||
- [ ] **Step 4: Commit** — `git commit -m "feat(secrets)!: retire the gRPC secrets hub in ScadaBridge — sites get secrets via Deploy Artifacts; rig central on the shared SQL store" -- src/ZB.MOM.WW.ScadaBridge.Host tests/ZB.MOM.WW.ScadaBridge.Host.Tests docker/docker-compose.yml Directory.Packages.props`
|
||||
|
||||
```json:metadata
|
||||
{"files": ["src/ZB.MOM.WW.ScadaBridge.Host/SecretsRegistration.cs", "src/ZB.MOM.WW.ScadaBridge.Host/Program.cs", "src/ZB.MOM.WW.ScadaBridge.Host/StartupValidator.cs", "docker/docker-compose.yml"], "verifyCommand": "dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests", "acceptanceCriteria": ["no hub wiring left in ScadaBridge", "Replication:Enabled=true fails startup with the retirement message", "wiring tests green", "compose renders; centrals SqlServer store; no hub env anywhere"], "modelTier": "standard"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 12: Documentation sync
|
||||
|
||||
**Goal:** The spec docs, CLAUDE.md, the deferred-work register and the umbrella index describe secrets-via-artifacts and the hub retirement.
|
||||
|
||||
**Files:**
|
||||
- Modify: `docs/requirements/Component-DeploymentManager.md` (artifact set + secrets, fail-closed, hash, audit), `Component-SiteRuntime.md` (deployed_secrets, mirror, options), `Component-Host.md` (Secrets section: providers, hub retired, bootstrap secrets out-of-band), `Component-CentralUI.md` (Sites page: secrets make a site out of date; failure messages)
|
||||
- Modify: `docs/deployment/topology-guide.md` (KEK must match on every central + site node; standby convergence interval)
|
||||
- Modify: `CLAUDE.md` (replace the secrets/hub facts with one bullet under *Templates & Deployment*; update the LocalDb bullet "ten tables" → eleven incl. `deployed_secrets`)
|
||||
- Modify: `docs/plans/2026-07-08-deferred-work-register.md` (new row: vd03 + QA rollout — zbsecrets store, `Secrets:Store:Provider=SqlServer` on both centrals, KEK alignment check, recreate hand-seeded site secrets centrally, Deploy Artifacts)
|
||||
- Modify: `../scadaproj/CLAUDE.md` (ScadaBridge row: short line; Secrets 0.6.3 note in the libraries table) and `../scadaproj/ZB.MOM.WW.Secrets/CLAUDE.md` (ScadaBridge no longer a hub consumer)
|
||||
|
||||
**Acceptance Criteria:**
|
||||
- [ ] `grep -rn "GrpcHub\|secrets hub" docs/requirements CLAUDE.md` returns only historical/retired mentions.
|
||||
- [ ] Register row lists the four rollout prerequisites and the trigger ("user asks for the vd03/QA secrets window").
|
||||
- [ ] Umbrella index row stays a few lines.
|
||||
|
||||
**Verify:** `grep -rn "deployed_secrets" docs/requirements/Component-SiteRuntime.md CLAUDE.md` → matches in both; `git diff --stat` shows only doc files.
|
||||
|
||||
**Steps:**
|
||||
|
||||
- [ ] **Step 1:** Edit the docs listed (quote the design doc sections rather than restating detail).
|
||||
- [ ] **Step 2:** `git diff` review. **Step 3: Commit** in each repo — `git commit -m "docs: site secrets ride the artifact deploy; gRPC hub retired in ScadaBridge" -- docs CLAUDE.md` and `git -C ../scadaproj commit -m "index: ScadaBridge secrets via artifact deploy; Secrets 0.6.3" -- CLAUDE.md ZB.MOM.WW.Secrets/CLAUDE.md`
|
||||
|
||||
```json:metadata
|
||||
{"files": ["docs/requirements/Component-DeploymentManager.md", "docs/requirements/Component-SiteRuntime.md", "docs/requirements/Component-Host.md", "CLAUDE.md", "docs/plans/2026-07-08-deferred-work-register.md"], "verifyCommand": "grep -rn deployed_secrets docs/requirements/Component-SiteRuntime.md CLAUDE.md", "acceptanceCriteria": ["component docs + topology guide updated", "CLAUDE.md bullet + LocalDb eleven tables", "register rollout row with 4 prerequisites", "umbrella + Secrets CLAUDE.md updated"], "modelTier": "mechanical"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 13: Full test run, rig rebuild, live gate
|
||||
|
||||
**Goal:** The whole solution is green and the rebuilt docker rig demonstrates secrets deploying with artifacts end to end, including standby convergence, staleness, removal, fail-closed and KEK mismatch.
|
||||
|
||||
**Files:**
|
||||
- Create: `docker/verify-site-secrets.sh` (idempotent gate script, prints PASS/FAIL per check, exits non-zero on any FAIL)
|
||||
- Create: `docs/plans/2026-09-28-site-secrets-artifact-deploy-live-gate.md` (results record)
|
||||
|
||||
**Acceptance Criteria:**
|
||||
- [ ] `dotnet test ZB.MOM.WW.ScadaBridge.slnx` green except the known env-only Playwright residue (list any failure explicitly).
|
||||
- [ ] `bash docker/deploy.sh` succeeds; all 8 nodes `/health/ready` 200.
|
||||
- [ ] Gate checks (each PASS):
|
||||
1. Create secret `sb-gate-mxgw` at central (Secrets CLI or `/admin/secrets`) and an unbound MxGateway data connection on site-a with `"ApiKey":"secret:sb-gate-mxgw"` → Sites page/`site list` shows site-a **Out of date**, site-b/site-c unchanged.
|
||||
2. `deploy artifacts --site-id <site-a>` → success, `secretsDeployed: 1`; site-a **Current**.
|
||||
3. Both site-a nodes' `secrets.db` contain `sb-gate-mxgw` at the central revision (`docker exec` + `sqlite3` on the library table, ciphertext compare by SHA-256) — the standby within 30 s.
|
||||
4. Rotate the secret at central → site-a **Out of date** → deploy → both nodes at the new revision.
|
||||
5. Remove the `secret:` reference → deploy → both nodes no longer hold the secret; an out-of-band secret seeded on site-a-a beforehand is still present.
|
||||
6. Reference a non-existent `sb-gate-missing` → deploy site-a fails with the named message; site-a stays **Out of date**; nothing changed on the site.
|
||||
7. Same connection on site-b (different dev KEK) → deploy fails with the KEK-mismatch message naming the KekId.
|
||||
8. Audit log has one `DeploySiteSecrets` row per secret-bearing deploy; `grep` of all 8 nodes' logs for the secret's plaintext and base64 ciphertext → 0 hits.
|
||||
- [ ] Gate script and results record committed; gate objects cleaned up by the script's teardown.
|
||||
|
||||
**Verify:** `bash docker/verify-site-secrets.sh` → `PASS 8/8`.
|
||||
|
||||
**Steps:**
|
||||
|
||||
- [ ] **Step 1:** `dotnet build ZB.MOM.WW.ScadaBridge.slnx && dotnet test ZB.MOM.WW.ScadaBridge.slnx` — record counts.
|
||||
- [ ] **Step 2:** `bash docker/deploy.sh` (from the primary checkout, not a worktree — see memory: deploy.sh from a worktree rehomes mounts); wait for readiness.
|
||||
- [ ] **Step 3:** Write `docker/verify-site-secrets.sh` implementing checks 1–8 with the CLI (`src/ZB.MOM.WW.ScadaBridge.CLI/bin/Debug/net10.0/scadabridge --url http://localhost:9000 --username multi-role --password password`), `docker exec`, and a trap-based teardown; run it until PASS 8/8, fixing product defects in their owning task's files (commit separately with the task tag).
|
||||
- [ ] **Step 4:** Write the results record (method, per-check evidence, timings incl. standby convergence), commit — `git commit -m "test(rig): site secrets artifact-deploy live gate PASS 8/8" -- docker/verify-site-secrets.sh docs/plans/2026-09-28-site-secrets-artifact-deploy-live-gate.md`
|
||||
|
||||
```json:metadata
|
||||
{"files": ["docker/verify-site-secrets.sh", "docs/plans/2026-09-28-site-secrets-artifact-deploy-live-gate.md"], "verifyCommand": "bash docker/verify-site-secrets.sh", "acceptanceCriteria": ["solution tests green (env-only residue listed)", "rig rebuilt, 8 nodes ready", "gate checks 1-8 PASS incl. standby convergence, removal, fail-closed, KEK mismatch, audit + no-material grep", "gate script + results record committed"], "modelTier": "standard"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Dependencies
|
||||
|
||||
- Task 1 is independent (library) — run first so the rig transition in Task 11/13 is observable.
|
||||
- Task 2 → Tasks 4, 6, 7, 9, 10. Task 3 → Task 6. Task 4 → Task 6. Task 5 → Task 6 (central store DI) and Task 11.
|
||||
- Task 6 → Task 7. Task 8 → Task 9 → Task 10.
|
||||
- Tasks 7 and 10 → Task 11 → Task 12 → Task 13.
|
||||
@@ -0,0 +1,120 @@
|
||||
{
|
||||
"planPath": "docs/plans/2026-09-28-site-secrets-artifact-deploy.md",
|
||||
"tasks": [
|
||||
{
|
||||
"id": 1,
|
||||
"subject": "Task 1: Secrets library 0.6.3 \u2014 hub sweep logging + status",
|
||||
"status": "pending",
|
||||
"blockedBy": [],
|
||||
"description": "**Goal:** Every hub sweep outcome is observable (Debug in-sync + hourly Information heartbeat + recovery line) and a public `ISecretsHubSyncStatus` exposes last sweep/success/failures; published as 0.6.3 and pinned in ScadaBridge.\n\n**Files:**\n- Modify: `~/Desktop/scadaproj/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.Grpc/GrpcSecretSyncService.cs`\n- Create: `~/Desktop/scadaproj/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.Grpc/ISecretsHubSyncStatus.cs`\n- Modify: `~/Desktop/scadaproj/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.Grpc/SecretsGrpcHubClientOptions.cs` (add `InSyncLogInterval`, default 1 h, validated > 0)\n- Modify: the Grpc hub client DI extension (register `ISecretsHubSyncStatus` singleton)\n- Modify: `~/Desktop/scadaproj/ZB.MOM.WW.Secrets/Directory.Build.props` (`<Version>0.6.3</Version>`), `~/Desktop/scadaproj/ZB.MOM.WW.Secrets/CLAUDE.md` (version narrative)\n- Test: `~/Desktop/scadaproj/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.Grpc.Tests/GrpcSecretSyncServiceLoggingTests.cs` (new)\n- Modify: `Directory.Packages.props` (ScadaBridge \u2014 all `ZB.MOM.WW.Secrets*` to 0.6.3)\n\n**Acceptance Criteria:**\n- [ ] An in-sync sweep (0 pulled) logs EventId 4 at Debug every time and EventId 5 `Secrets hub in sync ({Count} secret(s)); last change pulled {LastPullUtc}` at Information at most once per `InSyncLogInterval`.\n- [ ] The first successful sweep after \u2265 1 failure logs EventId 6 `Secrets hub sync recovered after {Failures} failed sweep(s)` at Information.\n- [ ] `ISecretsHubSyncStatus` returns `LastSweepUtc`, `LastSuccessUtc`, `ConsecutiveFailures`, `LastError` (message only, never a secret value) reflecting the last sweep.\n- [ ] Existing EventIds 1\u20133 unchanged; library tests green; package 0.6.3 on the Gitea feed; ScadaBridge builds on 0.6.3.\n\n**Verify:** `cd ~/Desktop/scadaproj/ZB.MOM.WW.Secrets && dotnet test --filter FullyQualifiedName~GrpcSecretSyncServiceLoggingTests` \u2192 all pass; `cd ~/Desktop/ScadaBridge && dotnet build ZB.MOM.WW.ScadaBridge.slnx` \u2192 0 errors.\n\n```json:metadata\n{\"files\": [\"~/Desktop/scadaproj/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.Grpc/GrpcSecretSyncService.cs\", \"~/Desktop/scadaproj/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.Grpc/ISecretsHubSyncStatus.cs\", \"Directory.Packages.props\"], \"verifyCommand\": \"cd ~/Desktop/scadaproj/ZB.MOM.WW.Secrets && dotnet test --filter FullyQualifiedName~GrpcSecretSyncServiceLoggingTests\", \"acceptanceCriteria\": [\"EventId 4 Debug per in-sync sweep, EventId 5 Information at most once per InSyncLogInterval\", \"EventId 6 recovered line after failures\", \"ISecretsHubSyncStatus reflects last sweep\", \"0.6.3 published and pinned; ScadaBridge builds\"], \"modelTier\": \"standard\"}\n```"
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"subject": "Task 2: Wire contract \u2014 `SecretArtifact`, `Secrets`, `Secret",
|
||||
"status": "pending",
|
||||
"blockedBy": [],
|
||||
"description": "**Goal:** `DeployArtifactsCommand` carries an optional ciphertext secret list and `ArtifactDeploymentResponse` an optional applied count, end to end over gRPC, backward compatible.\n\n**Files:**\n- Create: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Artifacts/SecretArtifact.cs`\n- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Artifacts/DeployArtifactsCommand.cs`, `ArtifactDeploymentResponse.cs`\n- Modify: `src/ZB.MOM.WW.ScadaBridge.Communication/Protos/site_command.proto` (fields per Global Constraints), vendored `src/ZB.MOM.WW.ScadaBridge.Communication/SiteCommandGrpc/*` via `docker/regen-proto.sh sitecommand`\n- Modify: `src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteCommandDtoMapper.cs` (`ToProto`/`FromProto` for command and response)\n- Test: `tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteCommandDtoMapperSecretsTests.cs` (new)\n\n**Acceptance Criteria:**\n- [ ] Command round-trip preserves every `SecretArtifact` field byte-for-byte; `Secrets == null` round-trips as null (field absent), empty list as empty.\n- [ ] Response round-trip: `SecretsApplied` null \u21c4 absent wrapper, `3` \u21c4 `3`.\n- [ ] A `DeployArtifactsCommandDto` serialized WITHOUT field 9 (old central) maps to `Secrets == null`.\n- [ ] No `<Protobuf>` item left active in the csproj.\n\n**Verify:** `dotnet test tests/ZB.MOM.WW.ScadaBridge.Communication.Tests --filter FullyQualifiedName~SiteCommandDtoMapperSecretsTests` \u2192 all pass.\n\n```json:metadata\n{\"files\": [\"src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Artifacts/SecretArtifact.cs\", \"src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Artifacts/DeployArtifactsCommand.cs\", \"src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Artifacts/ArtifactDeploymentResponse.cs\", \"src/ZB.MOM.WW.ScadaBridge.Communication/Protos/site_command.proto\", \"src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteCommandDtoMapper.cs\"], \"verifyCommand\": \"dotnet test tests/ZB.MOM.WW.ScadaBridge.Communication.Tests --filter FullyQualifiedName~SiteCommandDtoMapperSecretsTests\", \"acceptanceCriteria\": [\"SecretArtifact round-trips byte-for-byte\", \"null Secrets stays null / absent\", \"SecretsApplied null and 3 round-trip\", \"no active Protobuf item committed\"], \"modelTier\": \"standard\"}\n```"
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"subject": "Task 3: `SecretReferenceScanner`",
|
||||
"status": "pending",
|
||||
"blockedBy": [],
|
||||
"description": "**Goal:** A pure function returning, per site, the distinct `secret:<name>` references found anywhere in its data connections' primary/backup JSON, each with the referencing connection names.\n\n**Files:**\n- Create: `src/ZB.MOM.WW.ScadaBridge.Commons/Types/Secrets/SecretReferenceScanner.cs`\n- Test: `tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Types/Secrets/SecretReferenceScannerTests.cs`\n\n**Acceptance Criteria:**\n- [ ] Finds `secret:` string values at any depth (objects and arrays), prefix case-insensitive, name trimmed.\n- [ ] Ignores non-string values, strings merely containing `secret:` mid-text, `${secret:X}` (node-local config form), empty names, and unparseable JSON (no throw).\n- [ ] Output ordinal-sorted by name; each entry lists referencing connections ordinal-sorted and de-duplicated.\n- [ ] Bootstrap names (`SB-GRPC-PSK-` prefix) are excluded.\n\n**Verify:** `dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests --filter FullyQualifiedName~SecretReferenceScannerTests` \u2192 all pass.\n\n```json:metadata\n{\"files\": [\"src/ZB.MOM.WW.ScadaBridge.Commons/Types/Secrets/SecretReferenceScanner.cs\", \"tests/ZB.MOM.WW.ScadaBridge.Commons.Tests/Types/Secrets/SecretReferenceScannerTests.cs\"], \"verifyCommand\": \"dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests --filter FullyQualifiedName~SecretReferenceScannerTests\", \"acceptanceCriteria\": [\"nested/case-insensitive refs found in primary+backup\", \"non-refs, ${secret:}, bad JSON, bootstrap names ignored\", \"sorted, de-duplicated output\"], \"modelTier\": \"mechanical\"}\n```"
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"subject": "Task 4: Secrets in the artifact revision hash",
|
||||
"status": "pending",
|
||||
"blockedBy": [
|
||||
2
|
||||
],
|
||||
"description": "**Goal:** `ArtifactRevisionHashService` folds the command's secrets (and missing-secret markers) into the hash; sites with no secret refs keep their existing hash.\n\n**Files:**\n- Modify: `src/ZB.MOM.WW.ScadaBridge.DeploymentManager/ArtifactRevisionHashService.cs`\n- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Artifacts/DeployArtifactsCommand.cs` (append `IReadOnlyList<string>? MissingSecrets = null` \u2014 central-only, never mapped to proto; document that on the parameter)\n- Test: `tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests/ArtifactRevisionHashServiceTests.cs`\n\n**Acceptance Criteria:**\n- [ ] A command with `Secrets == null` and `MissingSecrets == null` hashes identically to the pre-change golden value (add a golden-hash test computed on `main` BEFORE the change).\n- [ ] Changing a secret's revision, ciphertext, wrapped DEK or KekId changes the hash; reordering secrets does not.\n- [ ] A missing name hashes differently from the same name present.\n- [ ] `HashableRecords_PropertiesDeclaredAlphabetically` covers `HashableSecret` and passes.\n\n**Verify:** `dotnet test tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests --filter FullyQualifiedName~ArtifactRevisionHashServiceTests` \u2192 all pass.\n\n```json:metadata\n{\"files\": [\"src/ZB.MOM.WW.ScadaBridge.DeploymentManager/ArtifactRevisionHashService.cs\", \"tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests/ArtifactRevisionHashServiceTests.cs\"], \"verifyCommand\": \"dotnet test tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests --filter FullyQualifiedName~ArtifactRevisionHashServiceTests\", \"acceptanceCriteria\": [\"golden pre-change hash unchanged when no secrets\", \"revision/ciphertext/wrapped DEK/KekId changes move the hash; order does not\", \"missing marker differs from present\", \"alphabetical-properties test passes\"], \"modelTier\": \"standard\"}\n```"
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"subject": "Task 5: Central shared secret store \u2014 `Secrets:Store:Provide",
|
||||
"status": "pending",
|
||||
"blockedBy": [],
|
||||
"description": "**Goal:** A Central node can use the shared SQL-Server secret store without the hub; a clustered Central on `Sqlite` warns.\n\n**Files:**\n- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/SecretsRegistration.cs`\n- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/StartupValidator.cs`\n- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/appsettings.json` (`\"Store\": { \"Provider\": \"Sqlite\" }` under `Secrets`, with an `_comment`)\n- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SecretsReplicationWiringTests.cs` (extend), `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/StartupValidatorTests.cs` (extend)\n\n**Acceptance Criteria:**\n- [ ] Central + `Provider=SqlServer` + connection string \u2192 `ISecretStore` resolves to the library's SQL-Server store; no hub endpoint mapped.\n- [ ] Central + `Provider=SqlServer` + blank connection string \u2192 startup failure naming `Secrets:SqlServer:ConnectionString`; a `${secret:` value is rejected (bootstrap rule, same message style as the existing Grpc-mode check).\n- [ ] Site role ignores `Secrets:Store:Provider` (always local SQLite).\n- [ ] Clustered Central (seed-nodes > 1) with `Provider=Sqlite` logs one Warning naming the per-node divergence risk; unknown provider value fails validation.\n\n**Verify:** `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter \"FullyQualifiedName~SecretsReplicationWiringTests|FullyQualifiedName~StartupValidatorTests\"` \u2192 all pass.\n\n```json:metadata\n{\"files\": [\"src/ZB.MOM.WW.ScadaBridge.Host/SecretsRegistration.cs\", \"src/ZB.MOM.WW.ScadaBridge.Host/StartupValidator.cs\", \"src/ZB.MOM.WW.ScadaBridge.Host/appsettings.json\"], \"verifyCommand\": \"dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter \\\"FullyQualifiedName~SecretsReplicationWiringTests|FullyQualifiedName~StartupValidatorTests\\\"\", \"acceptanceCriteria\": [\"Central+SqlServer resolves shared SQL store, no hub\", \"blank or ${secret:} connection string fails naming the key\", \"Site ignores provider\", \"clustered Central on Sqlite warns; unknown provider fails\"], \"modelTier\": \"standard\"}\n```"
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"subject": "Task 6: Central \u2014 collect a site's secrets, fail closed, log",
|
||||
"status": "pending",
|
||||
"blockedBy": [
|
||||
2,
|
||||
3,
|
||||
4,
|
||||
5
|
||||
],
|
||||
"description": "**Goal:** `ArtifactDeploymentService` adds each site's referenced secrets (ciphertext rows from `ISecretStore`) to its command, fails that site closed on a missing secret, logs names/revisions, and `ComputeCurrentArtifactHashesAsync` includes missing markers.\n\n**Files:**\n- Create: `src/ZB.MOM.WW.ScadaBridge.DeploymentManager/SiteSecretsCollector.cs`\n- Modify: `src/ZB.MOM.WW.ScadaBridge.DeploymentManager/ArtifactDeploymentService.cs` (`BuildDeployArtifactsCommandAsync`, `DeployCoreAsync`, `RetryForSiteAsync`, `ComputeCurrentArtifactHashesAsync`)\n- Modify: `src/ZB.MOM.WW.ScadaBridge.DeploymentManager/ZB.MOM.WW.ScadaBridge.DeploymentManager.csproj` (`<PackageReference Include=\"ZB.MOM.WW.Secrets.Abstractions\" />`), DeploymentManager DI registration (`SiteSecretsCollector` singleton)\n- Test: `tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests/SiteSecretsCollectorTests.cs` (new), `ArtifactDeploymentServiceTests.cs` (extend)\n\n**Acceptance Criteria:**\n- [ ] A site whose data connections reference `A` and `B` gets a command whose `Secrets` are exactly A and B's `StoredSecret` rows mapped field-for-field; other central secrets are not included.\n- [ ] A referenced name that is absent or `IsDeleted` \u2192 that site's `SiteArtifactResult` fails with `Secret 'X' referenced by data connection(s) C1, C2 is not in the central secret store`, nothing is sent to that site, other sites in the batch still deploy.\n- [ ] `ComputeCurrentArtifactHashesAsync` puts missing names in `MissingSecrets` instead of failing, so such a site reads *Out of date*.\n- [ ] Central logs Information `Deploying {Count} secret(s) to site {SiteId}: {Secrets}` (formatted `name@rN`) and Error for missing; a test asserts no log/exception text contains the secret's plaintext or base64 ciphertext.\n\n**Verify:** `dotnet test tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests --filter \"FullyQualifiedName~SiteSecretsCollectorTests|FullyQualifiedName~ArtifactDeploymentServiceTests\"` \u2192 all pass.\n\n```json:metadata\n{\"files\": [\"src/ZB.MOM.WW.ScadaBridge.DeploymentManager/SiteSecretsCollector.cs\", \"src/ZB.MOM.WW.ScadaBridge.DeploymentManager/ArtifactDeploymentService.cs\"], \"verifyCommand\": \"dotnet test tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests --filter \\\"FullyQualifiedName~SiteSecretsCollectorTests|FullyQualifiedName~ArtifactDeploymentServiceTests\\\"\", \"acceptanceCriteria\": [\"only referenced rows shipped, field-for-field\", \"missing/tombstoned ref fails only that site with the named message, nothing sent\", \"current-hash uses missing markers\", \"logs name@revision, never material\"], \"modelTier\": \"standard\"}\n```"
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"subject": "Task 7: Central \u2014 stamp gating for old sites + audit row",
|
||||
"status": "pending",
|
||||
"blockedBy": [
|
||||
6
|
||||
],
|
||||
"description": "**Goal:** Central never stamps a site *Current* unless the site confirmed applying every shipped secret, and every secret deployment writes one audit row.\n\n**Files:**\n- Modify: `src/ZB.MOM.WW.ScadaBridge.DeploymentManager/ArtifactDeploymentService.cs`\n- Modify: `SiteArtifactResult` record (same file or its Commons location) \u2014 append `int? SecretsDeployed = null`\n- Modify: `src/ZB.MOM.WW.ScadaBridge.CLI/Commands/DeployCommands.cs` only if its table output enumerates columns (JSON output carries the new field automatically)\n- Test: `tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests/ArtifactDeploymentServiceTests.cs`\n\n**Acceptance Criteria:**\n- [ ] Command with \u2265 1 secret + response `Success=true, SecretsApplied=null` \u2192 site result is a failure `Site build predates secret deployment \u2014 upgrade the site before deploying secrets`; hash NOT stamped.\n- [ ] `SecretsApplied` \u2260 shipped count \u2192 failure `Site applied {n} of {m} secrets`; not stamped.\n- [ ] Command with no secrets \u2192 old-site response (`SecretsApplied=null`) still succeeds and stamps (unchanged behaviour).\n- [ ] One `IAuditService.LogAsync(user, \"DeploySiteSecrets\", \"SiteSecrets\", siteIdentifier, siteName, { deploymentId, secrets:[{name,revision}], missing:[name], success, error })` per site whose command carried or would have carried secrets (incl. the fail-closed case); none for sites with no refs. Audit has no material.\n\n**Verify:** `dotnet test tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests --filter FullyQualifiedName~ArtifactDeploymentServiceTests` \u2192 all pass.\n\n```json:metadata\n{\"files\": [\"src/ZB.MOM.WW.ScadaBridge.DeploymentManager/ArtifactDeploymentService.cs\"], \"verifyCommand\": \"dotnet test tests/ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests --filter FullyQualifiedName~ArtifactDeploymentServiceTests\", \"acceptanceCriteria\": [\"old site + secrets -> failure, not stamped\", \"partial apply -> failure, not stamped\", \"no secrets -> unchanged behaviour\", \"one DeploySiteSecrets audit row per secret-bearing site, no material\"], \"modelTier\": \"standard\"}\n```"
|
||||
},
|
||||
{
|
||||
"id": 8,
|
||||
"subject": "Task 8: Site storage \u2014 `deployed_secrets` (replicated) + `mi",
|
||||
"status": "pending",
|
||||
"blockedBy": [
|
||||
2
|
||||
],
|
||||
"description": "**Goal:** The site database has a replicated ciphertext table for deployed secrets and a node-local ledger of mirrored names, with full-set accessors.\n\n**Files:**\n- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageSchema.cs`\n- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs`\n- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs` (`\"deployed_secrets\"` appended to `ReplicatedTables`; update the \"ten tables\" comments to eleven; `mirrored_secrets` deliberately NOT registered, with a comment why)\n- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/DeployedSecretsStorageTests.cs` (new); existing `SiteLocalDbReplicatedTableFkGuardTests` must still pass\n\n**Acceptance Criteria:**\n- [ ] `UpsertDeployedSecretsAsync(IReadOnlyList<SecretArtifact>, deploymentId)` upserts rows (`is_removed = 0`) and marks every other row `is_removed = 1`; returns `(applied names, newly removed names)`.\n- [ ] `GetDeployedSecretsAsync()` returns all rows incl. removed with their `SecretArtifact` round-tripped from `row_json`.\n- [ ] `mirrored_secrets` get/add/remove work; it is not in `ReplicatedTables`.\n- [ ] A security test asserts `row_json` holds only base64 ciphertext fields (deserializes to `SecretArtifact`, no extra properties).\n- [ ] FK guard test green (no FKs on either table).\n\n**Verify:** `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter FullyQualifiedName~DeployedSecretsStorageTests` and `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests --filter FullyQualifiedName~SiteLocalDbReplicatedTableFkGuardTests` \u2192 all pass.\n\n```json:metadata\n{\"files\": [\"src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageSchema.cs\", \"src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs\", \"src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs\"], \"verifyCommand\": \"dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter FullyQualifiedName~DeployedSecretsStorageTests\", \"acceptanceCriteria\": [\"full-set upsert marks absent rows removed\", \"rows round-trip SecretArtifact\", \"mirrored_secrets local only\", \"row_json ciphertext-only security test\", \"FK guard green\"], \"modelTier\": \"standard\"}\n```"
|
||||
},
|
||||
{
|
||||
"id": 9,
|
||||
"subject": "Task 9: `DeployedSecretsMirror` \u2014 per-node apply + verify",
|
||||
"status": "pending",
|
||||
"blockedBy": [
|
||||
8
|
||||
],
|
||||
"description": "**Goal:** Every site node reconciles `deployed_secrets` into its own `secrets.db` (apply live rows, delete removed rows it previously mirrored), verifies each deployed secret decrypts, logs changes, and can be invoked synchronously.\n\n**Files:**\n- Create: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Secrets/DeployedSecretsMirror.cs` (service + `IHostedService` loop)\n- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs` (+ `DeployedSecretsMirrorInterval`, default 15 s) and `SiteRuntimeOptionsValidator.cs` (> 0)\n- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ZB.MOM.WW.ScadaBridge.SiteRuntime.csproj` (`ZB.MOM.WW.Secrets.Abstractions`), SiteRuntime DI (`AddSingleton<DeployedSecretsMirror>()` + `AddHostedService(sp => sp.GetRequiredService<DeployedSecretsMirror>())`)\n- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Secrets/DeployedSecretsMirrorTests.cs`\n\n**Acceptance Criteria:**\n- [ ] `ReconcileAsync()` applies each live row with `ISecretStore.ApplyReplicatedAsync` (mapped back to `StoredSecret`), records it in `mirrored_secrets`, and returns a `MirrorResult(Applied, Removed, Failed)`.\n- [ ] A removed row is deleted from the store only if its name is in `mirrored_secrets`; a same-named out-of-band secret never mirrored is left alone.\n- [ ] Each live secret is trial-resolved via `ISecretResolver`; a failure (null or exception) is a `Failed` entry `Secret '{Name}' (kek {KekId}) did not decrypt on this node \u2014 the node's master key differs from central's` and is logged at Error.\n- [ ] Logging: Information per applied/removed secret and one summary when anything changed; Debug when in sync; never values.\n- [ ] The hosted loop runs once at start and every `DeployedSecretsMirrorInterval`; a reconcile exception is logged and the loop continues.\n\n**Verify:** `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter FullyQualifiedName~DeployedSecretsMirrorTests` \u2192 all pass.\n\n```json:metadata\n{\"files\": [\"src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Secrets/DeployedSecretsMirror.cs\", \"src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs\", \"src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptionsValidator.cs\"], \"verifyCommand\": \"dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter FullyQualifiedName~DeployedSecretsMirrorTests\", \"acceptanceCriteria\": [\"live rows applied via ApplyReplicatedAsync and ledgered\", \"removed rows deleted only if previously mirrored\", \"decrypt failure reported with KekId at Error\", \"change/in-sync logging without values\", \"hosted loop at start + interval, survives exceptions\"], \"modelTier\": \"standard\"}\n```"
|
||||
},
|
||||
{
|
||||
"id": 10,
|
||||
"subject": "Task 10: Site apply \u2014 secrets in `HandleDeployArtifacts`",
|
||||
"status": "pending",
|
||||
"blockedBy": [
|
||||
9
|
||||
],
|
||||
"description": "**Goal:** The active site node writes shipped secrets to `deployed_secrets`, runs the mirror synchronously, and answers with `SecretsApplied` or a failure.\n\n**Files:**\n- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs` (`HandleDeployArtifacts`, constructor gets `DeployedSecretsMirror?`)\n- Modify: the actor's Props/DI construction site to pass the mirror\n- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorArtifactSecretsTests.cs` (new; follow the existing artifact-apply actor tests)\n\n**Acceptance Criteria:**\n- [ ] `Secrets == null` \u2192 `deployed_secrets` untouched, response `SecretsApplied = null` (old-central compatible).\n- [ ] `Secrets` present (incl. empty) \u2192 full-set write, synchronous `ReconcileAsync`, response `Success=true, SecretsApplied = shipped count` when the mirror reports no failures.\n- [ ] Any mirror failure \u2192 `Success=false`, `ErrorMessage` = the failure lines joined with `; `, `SecretsApplied` = count applied without failure.\n- [ ] Site Information log `Applying {Count} deployed secret(s), deploymentId={DeploymentId}` + the mirror's per-secret lines.\n\n**Verify:** `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter FullyQualifiedName~DeploymentManagerActorArtifactSecretsTests` \u2192 all pass.\n\n```json:metadata\n{\"files\": [\"src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs\"], \"verifyCommand\": \"dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter FullyQualifiedName~DeploymentManagerActorArtifactSecretsTests\", \"acceptanceCriteria\": [\"null Secrets leaves table and SecretsApplied null\", \"secrets -> full-set write + sync mirror + SecretsApplied=count\", \"mirror failure -> Success=false with KekId message\", \"apply logging\"], \"modelTier\": \"standard\"}\n```"
|
||||
},
|
||||
{
|
||||
"id": 11,
|
||||
"subject": "Task 11: Retire the gRPC hub for ScadaBridge; switch the rig",
|
||||
"status": "pending",
|
||||
"blockedBy": [
|
||||
7,
|
||||
10,
|
||||
1
|
||||
],
|
||||
"description": "**Goal:** ScadaBridge no longer composes the secrets hub (client or server); config asking for it fails fast; the docker rig's central pair uses `Secrets:Store:Provider=SqlServer` and site-a gets secrets only via artifact deploy.\n\n**Files:**\n- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/SecretsRegistration.cs` (delete the Grpc hub/hub-client and SqlServer-replication branches; keep local SQLite + Task 5's shared store), `Program.cs` (remove `MapScadaBridgeSecretsHub`), `StartupValidator.cs` (reject `Secrets:Replication:Enabled=true`: `Secrets replication (the gRPC hub) is retired in ScadaBridge \u2014 sites receive secrets with Deploy Artifacts; central uses Secrets:Store:Provider=SqlServer`)\n- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/appsettings.json` (remove `Replication`/`GrpcHub` blocks; keep `SqlServer.ConnectionString`/`SchemaName`; update comments)\n- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/ZB.MOM.WW.ScadaBridge.Host.csproj` (drop `ZB.MOM.WW.Secrets.Replicator.Grpc` reference) and `Directory.Packages.props` if nothing else uses it\n- Modify: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SecretsReplicationWiringTests.cs`, `MappedGrpcAuthenticationCoverageTests.cs` (hub route gone)\n- Modify: `docker/docker-compose.yml` (delete `x-secrets-hub-env` + `x-secrets-hub-site-env` anchors and their merges; central-a/b get `Secrets__Store__Provider: \"SqlServer\"` + the existing `x-secrets-hub-central-env` connection string renamed `x-secrets-central-store-env`; the rig KEK `ZB_SECRETS_MASTER_KEY` goes on central-a/b AND site-a-a/b \u2014 every node that handles deployed secrets must share it; site-b/site-c keep a DIFFERENT dev KEK so the Task 13 mismatch negative is real)\n\n**Acceptance Criteria:**\n- [ ] No ScadaBridge code references `AddZbSecretsGrpcHub`, `AddZbSecretsGrpcHubClient` or `MapScadaBridgeSecretsHub`.\n- [ ] `Secrets:Replication:Enabled=true` on either role \u2192 startup failure with the message above.\n- [ ] Wiring tests cover: Central Sqlite, Central SqlServer store, Site local \u2014 all green; hub-route auth coverage test updated.\n- [ ] `docker compose config` renders; central-a/b carry `Secrets__Store__Provider=SqlServer`; no node carries `Secrets__Replication__*` or `Secrets__GrpcHub__*`.\n\n**Verify:** `dotnet build ZB.MOM.WW.ScadaBridge.slnx` \u2192 0 errors; `dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests` \u2192 all pass; `docker compose -f docker/docker-compose.yml config | grep -c Secrets__Replication` \u2192 `0`.\n\n```json:metadata\n{\"files\": [\"src/ZB.MOM.WW.ScadaBridge.Host/SecretsRegistration.cs\", \"src/ZB.MOM.WW.ScadaBridge.Host/Program.cs\", \"src/ZB.MOM.WW.ScadaBridge.Host/StartupValidator.cs\", \"docker/docker-compose.yml\"], \"verifyCommand\": \"dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests\", \"acceptanceCriteria\": [\"no hub wiring left in ScadaBridge\", \"Replication:Enabled=true fails startup with the retirement message\", \"wiring tests green\", \"compose renders; centrals SqlServer store; no hub env anywhere\"], \"modelTier\": \"standard\"}\n```"
|
||||
},
|
||||
{
|
||||
"id": 12,
|
||||
"subject": "Task 12: Documentation sync",
|
||||
"status": "pending",
|
||||
"blockedBy": [
|
||||
11
|
||||
],
|
||||
"description": "**Goal:** The spec docs, CLAUDE.md, the deferred-work register and the umbrella index describe secrets-via-artifacts and the hub retirement.\n\n**Files:**\n- Modify: `docs/requirements/Component-DeploymentManager.md` (artifact set + secrets, fail-closed, hash, audit), `Component-SiteRuntime.md` (deployed_secrets, mirror, options), `Component-Host.md` (Secrets section: providers, hub retired, bootstrap secrets out-of-band), `Component-CentralUI.md` (Sites page: secrets make a site out of date; failure messages)\n- Modify: `docs/deployment/topology-guide.md` (KEK must match on every central + site node; standby convergence interval)\n- Modify: `CLAUDE.md` (replace the secrets/hub facts with one bullet under *Templates & Deployment*; update the LocalDb bullet \"ten tables\" \u2192 eleven incl. `deployed_secrets`)\n- Modify: `docs/plans/2026-07-08-deferred-work-register.md` (new row: vd03 + QA rollout \u2014 zbsecrets store, `Secrets:Store:Provider=SqlServer` on both centrals, KEK alignment check, recreate hand-seeded site secrets centrally, Deploy Artifacts)\n- Modify: `../scadaproj/CLAUDE.md` (ScadaBridge row: short line; Secrets 0.6.3 note in the libraries table) and `../scadaproj/ZB.MOM.WW.Secrets/CLAUDE.md` (ScadaBridge no longer a hub consumer)\n\n**Acceptance Criteria:**\n- [ ] `grep -rn \"GrpcHub\\|secrets hub\" docs/requirements CLAUDE.md` returns only historical/retired mentions.\n- [ ] Register row lists the four rollout prerequisites and the trigger (\"user asks for the vd03/QA secrets window\").\n- [ ] Umbrella index row stays a few lines.\n\n**Verify:** `grep -rn \"deployed_secrets\" docs/requirements/Component-SiteRuntime.md CLAUDE.md` \u2192 matches in both; `git diff --stat` shows only doc files.\n\n```json:metadata\n{\"files\": [\"docs/requirements/Component-DeploymentManager.md\", \"docs/requirements/Component-SiteRuntime.md\", \"docs/requirements/Component-Host.md\", \"CLAUDE.md\", \"docs/plans/2026-07-08-deferred-work-register.md\"], \"verifyCommand\": \"grep -rn deployed_secrets docs/requirements/Component-SiteRuntime.md CLAUDE.md\", \"acceptanceCriteria\": [\"component docs + topology guide updated\", \"CLAUDE.md bullet + LocalDb eleven tables\", \"register rollout row with 4 prerequisites\", \"umbrella + Secrets CLAUDE.md updated\"], \"modelTier\": \"mechanical\"}\n```"
|
||||
},
|
||||
{
|
||||
"id": 13,
|
||||
"subject": "Task 13: Full test run, rig rebuild, live gate",
|
||||
"status": "pending",
|
||||
"blockedBy": [
|
||||
12
|
||||
],
|
||||
"description": "**Goal:** The whole solution is green and the rebuilt docker rig demonstrates secrets deploying with artifacts end to end, including standby convergence, staleness, removal, fail-closed and KEK mismatch.\n\n**Files:**\n- Create: `docker/verify-site-secrets.sh` (idempotent gate script, prints PASS/FAIL per check, exits non-zero on any FAIL)\n- Create: `docs/plans/2026-09-28-site-secrets-artifact-deploy-live-gate.md` (results record)\n\n**Acceptance Criteria:**\n- [ ] `dotnet test ZB.MOM.WW.ScadaBridge.slnx` green except the known env-only Playwright residue (list any failure explicitly).\n- [ ] `bash docker/deploy.sh` succeeds; all 8 nodes `/health/ready` 200.\n- [ ] Gate checks (each PASS):\n 1. Create secret `sb-gate-mxgw` at central (Secrets CLI or `/admin/secrets`) and an unbound MxGateway data connection on site-a with `\"ApiKey\":\"secret:sb-gate-mxgw\"` \u2192 Sites page/`site list` shows site-a **Out of date**, site-b/site-c unchanged.\n 2. `deploy artifacts --site-id <site-a>` \u2192 success, `secretsDeployed: 1`; site-a **Current**.\n 3. Both site-a nodes' `secrets.db` contain `sb-gate-mxgw` at the central revision (`docker exec` + `sqlite3` on the library table, ciphertext compare by SHA-256) \u2014 the standby within 30 s.\n 4. Rotate the secret at central \u2192 site-a **Out of date** \u2192 deploy \u2192 both nodes at the new revision.\n 5. Remove the `secret:` reference \u2192 deploy \u2192 both nodes no longer hold the secret; an out-of-band secret seeded on site-a-a beforehand is still present.\n 6. Reference a non-existent `sb-gate-missing` \u2192 deploy site-a fails with the named message; site-a stays **Out of date**; nothing changed on the site.\n 7. Same connection on site-b (different dev KEK) \u2192 deploy fails with the KEK-mismatch message naming the KekId.\n 8. Audit log has one `DeploySiteSecrets` row per secret-bearing deploy; `grep` of all 8 nodes' logs for the secret's plaintext and base64 ciphertext \u2192 0 hits.\n- [ ] Gate script and results record committed; gate objects cleaned up by the script's teardown.\n\n**Verify:** `bash docker/verify-site-secrets.sh` \u2192 `PASS 8/8`.\n\n```json:metadata\n{\"files\": [\"docker/verify-site-secrets.sh\", \"docs/plans/2026-09-28-site-secrets-artifact-deploy-live-gate.md\"], \"verifyCommand\": \"bash docker/verify-site-secrets.sh\", \"acceptanceCriteria\": [\"solution tests green (env-only residue listed)\", \"rig rebuilt, 8 nodes ready\", \"gate checks 1-8 PASS incl. standby convergence, removal, fail-closed, KEK mismatch, audit + no-material grep\", \"gate script + results record committed\"], \"modelTier\": \"standard\"}\n```"
|
||||
}
|
||||
],
|
||||
"lastUpdated": "2026-09-28T20:58:11.605203Z"
|
||||
}
|
||||
@@ -109,7 +109,7 @@ Central cluster only. Sites have no user interface.
|
||||
- **Move data connection between sites** (M9/T24): a **Move…** action on a data connection opens `MoveDataConnectionDialog`. The underlying `MoveDataConnectionCommand` guards against: target site does not exist, name collision at the target, any `InstanceConnectionBinding` referencing the connection, and any `InstanceNativeAlarmSourceOverride.ConnectionNameOverride` reference at the source site's templates. Blocked connections display the blocking reason.
|
||||
- **Server certificate management** (`/design/connections/{id}/certificates`, Admin role): a per-connection page that lists the contents of the site's OPC UA trusted-peer and rejected certificate stores (Subject / Issuer / Thumbprint / validity / Trusted-or-Rejected status) with a **Remove** action. The page makes clear the store is **node-wide for the site** (shared by every site node), not per data connection — trusting or removing a certificate affects all OPC UA connections at that site. Trust and Remove are central commands relayed to **both** site nodes so the node-local PKI stores stay consistent across failover (see Component-SiteRuntime.md, Component-DataConnectionLayer.md).
|
||||
- The site detail page exposes a new **"Audit feed"** tab that hosts the Audit Log page pre-filtered to `Site = <site>` — an in-context view of every operational audit event for that site.
|
||||
- **Artifacts badge (2026-09-14, design `docs/plans/2026-09-14-pending-updates-design.md`).** Each site card carries an **Artifacts** badge next to the identifier, one of three states: `Current` (light — the site's `Site.ArtifactRevisionHash` matches what a deploy would send today), `Out of date` (warning, `bi-exclamation-triangle`, tooltip = the last successful deploy's UTC timestamp), or `Never deployed` (secondary — the site has no stamp at all, including every site immediately after the migration that added the columns, until one artifact deploy). Below the badge a muted `N of M instances out of date` line links to `/deployment/topology` whenever the site has any deployed instances, since the same fleet snapshot already carries the count.
|
||||
- **Artifacts badge (2026-09-14, design `docs/plans/2026-09-14-pending-updates-design.md`).** Each site card carries an **Artifacts** badge next to the identifier, one of three states: `Current` (light — the site's `Site.ArtifactRevisionHash` matches what a deploy would send today), `Out of date` (warning, `bi-exclamation-triangle`, tooltip = the last successful deploy's UTC timestamp), or `Never deployed` (secondary — the site has no stamp at all, including every site immediately after the migration that added the columns, until one artifact deploy). Below the badge a muted `N of M instances out of date` line links to `/deployment/topology` whenever the site has any deployed instances, since the same fleet snapshot already carries the count. **A site's secrets count toward the same badge (2026-09-28, `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`)**: a data connection referencing a `secret:<name>` not yet created centrally hashes as `Missing` and reads `Out of date`, never `Current`, until the secret exists and a deploy ships it.
|
||||
- The page header gains a **Re-check** button (forces a fresh fleet evaluation, disabled while running) and an **"Updates as of HH:mm:ss UTC"** caption next to the page title. The snapshot loads **after** the site list, never blocking it — cards render immediately with a *checking…* placeholder for the badge/line, and a snapshot failure renders **unavailable** on the badge rather than hiding it or failing the page. A successful **Deploy Artifacts** (single-site or all-sites) forces a refresh of the snapshot so the badge flips to `Current` immediately rather than waiting out the cache TTL.
|
||||
|
||||
### Schema Library (Design Role)
|
||||
@@ -189,6 +189,7 @@ Central cluster only. Sites have no user interface.
|
||||
- **Deploy all**: A bulk action deploys artifacts to all sites at once.
|
||||
- This is a **separate action** from instance deployment — system-wide artifacts are not automatically pushed when definitions change.
|
||||
- **Track per-site deployment status**, backed by `Site.ArtifactRevisionHash` / `ArtifactsDeployedAt` (Component-DeploymentManager.md → *Per-site artifact revision*), stamped **only on a successful deploy to that site**. A failed site keeps its previous stamp (or none) and keeps reading out of date on the Sites page; after an upgrade that ships the stamp columns, every site reads *Never deployed* until one artifact deploy runs — see the Artifacts badge above.
|
||||
- **Secrets (2026-09-28).** A deploy carries only the secrets that site's data connections reference (Component-DeploymentManager.md → *Site secrets*); no separate UI surface picks them. A site referencing a secret that is missing or tombstoned centrally fails **that site's** deploy (others in the same "Deploy all" sweep are unaffected) with a message naming the secret and the referencing data connection(s) — e.g. `Secret 'X' referenced by data connection(s) A, B is not in the central secret store` — surfaced on the deployment result exactly like any other per-site artifact-deploy failure. A site whose build predates secret deployment, or that applied only some of the shipped secrets, likewise fails and is not stamped `Current`; both cases resolve by upgrading the site and redeploying.
|
||||
|
||||
### Debug View (Deployment Role)
|
||||
- Select a deployed instance and open a live debug view — the instance picked through the same shared **`InstancePicker`** combobox as the Flow Debugger, with an `ItemLabel` suffix of `(State)` on each row.
|
||||
|
||||
@@ -281,6 +281,9 @@ Component-NotificationService.md). Notification delivery happens on the central
|
||||
cluster, so no notification artifact or SMTP credential is ever distributed to
|
||||
sites.
|
||||
|
||||
A site's **secrets** are a fifth, *derived* artifact class riding the same command —
|
||||
see *Site secrets* below.
|
||||
|
||||
### Per-site artifact revision (2026-09-14)
|
||||
|
||||
`Site` carries two nullable columns — `ArtifactRevisionHash` (`string?`, max 100)
|
||||
@@ -331,6 +334,83 @@ design**: a partial map would silently report the unqueried sites as up to date,
|
||||
so the pending-updates cache instead discards the faulted flight and the UI shows
|
||||
the aggregate as unavailable.
|
||||
|
||||
### Site secrets (2026-09-28)
|
||||
|
||||
Site secrets ride the artifact deploy instead of the (now retired) gRPC secrets
|
||||
hub — design `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`; see
|
||||
that doc's *Implementation notes (as built)* section for the deviations called
|
||||
out below.
|
||||
|
||||
- **Derived, not configured.** `SiteSecretsCollector` scans the site's data
|
||||
connection artifacts (`SecretReferenceScanner` over `PrimaryConfigurationJson`
|
||||
/ `BackupConfigurationJson`) for `secret:<name>` string values and resolves
|
||||
each name against the central `ISecretStore`, producing the ciphertext-only
|
||||
`SecretArtifact` rows the command carries (`DeployArtifactsCommand.Secrets`,
|
||||
proto field 9) plus the set of unresolved references. A site with no `secret:`
|
||||
references ships no secrets and is unaffected by any of this.
|
||||
- **Fail closed, per site.** A site with one or more unresolved (absent or
|
||||
tombstoned) references fails its deploy *before* `_communicationService` is
|
||||
called — nothing is sent, the site keeps its previous stamp, and the error
|
||||
names the secret and the referencing data connection(s): `Secret 'X'
|
||||
referenced by data connection(s) A, B is not in the central secret store`
|
||||
(semicolon-joined for more than one). Other sites in the same sweep are
|
||||
unaffected.
|
||||
- **Stamp gating.** A site is stamped `ArtifactRevisionHash`/`ArtifactsDeployedAt`
|
||||
only when every shipped secret actually applied. Two failure shapes are
|
||||
distinguished from an ordinary site failure and never stamped: an **old
|
||||
site** (wire `Success = true`, secrets were shipped, but the response has no
|
||||
`SecretsApplied` at all — a pre-feature site build) fails with `Site build
|
||||
predates secret deployment — upgrade the site before deploying secrets`; a
|
||||
**partial apply** (`SecretsApplied` less than the shipped count) fails with
|
||||
`Site applied {n} of {m} secrets`. Both apply identically on the all-sites
|
||||
sweep (`DeployCoreAsync`) and the per-site retry (`RetryForSiteAsync`).
|
||||
- **Staleness hash.** `ArtifactRevisionHashService` folds secrets into the same
|
||||
hash as every other artifact class (`HashableSecret`: a SHA-256 digest of the
|
||||
ciphertext/nonce/tag/wrapped-DEK material, never plaintext, plus `KekId` and
|
||||
`Revision`) so a KEK rewrap correctly flips a site to out of date. A
|
||||
referenced-but-not-yet-created secret hashes as an explicit `Missing` marker
|
||||
(`DeployArtifactsCommand.MissingSecrets`, central-only — never on the wire),
|
||||
so that site reads *Out of date*, never *Current*, until the secret exists.
|
||||
**A `null` secrets list and an empty one hash identically** (the existing
|
||||
null-when-empty convention for every artifact class), so a site with no
|
||||
secret references keeps its pre-existing hash across the upgrade that added
|
||||
this feature.
|
||||
- **Always a list on the wire.** `DeployArtifactsCommand.Secrets` is never
|
||||
`null` from a secrets-aware central — a site with no secret references gets an
|
||||
**empty** list. The site applies it as the full set (every `deployed_secrets`
|
||||
row absent from it is soft-removed and the mirror deletes it on both nodes),
|
||||
whereas `null` is the pre-feature "central shipped no secrets" contract that
|
||||
touches nothing; shipping `null` once the last reference was dropped left the
|
||||
secret live on both site nodes indefinitely (caught by the Task 13 live gate,
|
||||
check 5). An old site ignores proto field 9, so the empty list costs it
|
||||
nothing, and the hash is unaffected (empty == null above).
|
||||
- **Audit.** One best-effort `DeploySiteSecrets` / `SiteSecrets` audit row is
|
||||
written per site deploy that carried secrets or a missing-secret failure
|
||||
(never for a site with no secret references at all) — after-state is `{
|
||||
deploymentId, secrets: [{name, revision}], missing: [name], success }`, which
|
||||
structurally excludes ciphertext. Written on both `DeployCoreAsync` and
|
||||
`RetryForSiteAsync`, including the fail-closed path (audited even though
|
||||
nothing was sent). The row carries `secrets` and `missing` only — there is
|
||||
**no `removed` field** (design §3.7 named one), and a deploy that only
|
||||
removes secrets (the site's last reference dropped, an empty list shipped)
|
||||
writes **no central row**; removals are logged on the site by
|
||||
`DeployedSecretsMirror`.
|
||||
- **Secret rotation and live connections.** The site writes and mirrors the
|
||||
shipped secrets before it pushes data connections to the DCL, so a
|
||||
connection created by the same deploy resolves a new secret on first
|
||||
connect; a rotation with unchanged connection config takes effect on that
|
||||
connection's next (re)connect — see Component-SiteRuntime.md → *Deployed
|
||||
Secrets*.
|
||||
- **One `secret:` parser.** `SecretReferenceScanner.TryParseSecretReference`
|
||||
(trimmed, case-insensitive prefix, trimmed name; `${secret:…}` never
|
||||
matches) is shared by the scanner and every adapter that resolves the
|
||||
reference (today `MxGatewayDataConnection`'s `ApiKey`), so the name central
|
||||
ships is exactly the name the site looks up.
|
||||
- **Central store.** Central resolves secrets from `ISecretStore`, composed by
|
||||
the Host per `Secrets:Store:Provider` — see Component-Host.md → *Secrets*.
|
||||
- **Site apply and the per-node mirror** — see Component-SiteRuntime.md →
|
||||
*Deployed Secrets*.
|
||||
|
||||
## Site-Side Apply Atomicity
|
||||
|
||||
Applying a deployment at the site is **all-or-nothing per instance**:
|
||||
@@ -420,6 +500,7 @@ The Deployment Manager sends the following commands to sites via the Communicati
|
||||
- **Configuration Database (MS SQL)**: Stores deployment status and deployed configuration snapshots.
|
||||
- **Security & Auth**: Enforces Deployment role (with optional site scoping).
|
||||
- **Configuration Database (via IAuditService)**: Logs all deployment actions, system-wide artifact deployments, and instance lifecycle changes.
|
||||
- **`ZB.MOM.WW.Secrets.Abstractions`** (`ISecretStore`, `SecretName`): `SiteSecretsCollector` resolves a site's referenced secrets against the Host-composed central store — see Component-Host.md → *Secrets*.
|
||||
|
||||
## Interactions
|
||||
|
||||
|
||||
@@ -249,6 +249,58 @@ reporting that it is a standby. The verdict remains fully observable through the
|
||||
|
||||
---
|
||||
|
||||
## Secrets (2026-09-28)
|
||||
|
||||
The Host composes `ZB.MOM.WW.Secrets` (a `scadaproj` shared library, not a ScadaBridge component —
|
||||
hence it has no row in the Component Registration Matrix below) via `SecretsRegistration.AddScadaBridgeSecrets`,
|
||||
called on both roles from their respective composition-root `Configure` methods. Design:
|
||||
`docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md` (see its *Implementation notes (as
|
||||
built)* section for where the shipped code deviates from the original design).
|
||||
|
||||
- **Site role — always a local, undecorated store.** A site node's `ISecretStore`/`ISecretResolver`
|
||||
is unconditionally the plain node-local SQLite store (`AddZbSecrets`), regardless of any
|
||||
`Secrets:Store:Provider` value — a Site never reads that key. This is a hard invariant, not
|
||||
just a default: `DeployedSecretsMirror` (Component-SiteRuntime.md) calls `ISecretStore.DeleteAsync`
|
||||
directly on the assumption that a delete is node-local; a replication-decorated store would have
|
||||
published that delete fleet-wide. Secrets reach a site only via Deploy Artifacts (see
|
||||
Component-DeploymentManager.md → *Site secrets*) — there is no other channel.
|
||||
- **Central role — `Secrets:Store:Provider`** (`Sqlite` default | `SqlServer`), read **only** on the
|
||||
Central role:
|
||||
- `Sqlite` (default): each central node keeps its own per-node SQLite secret store. On a
|
||||
**clustered** central (≥ 2 seed nodes) this is a documented misconfiguration — `StartupValidator`
|
||||
logs a boot-time Warning (same shape as the inbound-API-key-store precedent, register row 108),
|
||||
because the two nodes would deploy different secret sets to sites depending on which is active.
|
||||
- `SqlServer`: both central nodes read and write **one shared SQL Server store**
|
||||
(`Secrets:SqlServer:ConnectionString`, a literal/env value — never a `${secret:}` reference,
|
||||
the same bootstrap-circularity rule the inbound-API-key shared store follows) — the `/admin/secrets`
|
||||
page edits it, and it is the store `SiteSecretsCollector` resolves from. The pre-host `${secret:}`
|
||||
expander (the Layer-A pass that resolves `${secret:X}` tokens inside `appsettings` itself, before
|
||||
the DI container is built) reads the **same** shared store when `Provider = SqlServer`, so a
|
||||
central node's boot-time config tokens and its running `ISecretResolver` never disagree about
|
||||
what a token resolves to.
|
||||
- A connection string that is blank, or is itself a `${secret:}` reference (a bootstrap circularity),
|
||||
fails fast naming the offending key.
|
||||
- **Bootstrap secrets are never deployed.** The site's gRPC PSK (`SB-GRPC-PSK-{siteId}`), the LocalDb
|
||||
replication key, and the secrets master key (KEK) are never data-connection-referenced and are
|
||||
never shipped through Deploy Artifacts — a site needs them *before* it can receive a deploy at all,
|
||||
and pushing a rotated one through the channel it authenticates would sever that channel. They stay
|
||||
out-of-band: service environment variables, a hand-seeded store, or a master-key file, exactly as
|
||||
before this feature. Every node — every central and every site — **must share the same KEK**;
|
||||
see `docs/deployment/topology-guide.md` → *Secrets: shared store, per-site deploy, and the KEK*
|
||||
for where that is (and is not) checked.
|
||||
- **The gRPC secrets hub is retired in ScadaBridge (2026-09-28).** ScadaBridge no longer composes
|
||||
`ZB.MOM.WW.Secrets.Replicator.Grpc` on either role, and the historical SQL-Server *replication*
|
||||
mode (`Secrets:Replication:Mode=SqlServer` — distinct from `Secrets:Store:Provider=SqlServer`
|
||||
above, which shares one store rather than replicating between two) is retired too. Setting
|
||||
`Secrets:Replication:Enabled` to anything other than absent/empty/`false` now fails
|
||||
`StartupValidator.Validate` outright, on **either** role, with a message pointing at Deploy
|
||||
Artifacts (sites) and `Secrets:Store:Provider=SqlServer` (a clustered central). A leftover
|
||||
`Secrets:Replication:Mode` or any `Secrets:GrpcHub:*` key (harmless once `Enabled` is unset, but
|
||||
dead configuration) logs a startup Warning naming both sections so they can be cleaned up. The
|
||||
library itself keeps the hub for other consumers; only ScadaBridge stops using it.
|
||||
|
||||
---
|
||||
|
||||
## Component Registration Matrix
|
||||
|
||||
| Component | Central | Site | DI (`AddXxx`) | Actors (`AddXxxActors`) | Endpoints (`MapXxx`) |
|
||||
|
||||
@@ -162,6 +162,13 @@ flowchart TD
|
||||
- Recompiles shared scripts and makes updated code available to all Script Actors.
|
||||
- **Reconciles deletions**: each non-null artifact list is the complete set for its class, so after upserting, any stored row absent from the set is deleted (the artifact was deleted centrally — there is no per-artifact delete command). A removed shared script is also unregistered from the compiled script library, and a removed data connection's live DCL connection actor is stopped. A `null` list means "field not shipped" and reconciles nothing.
|
||||
|
||||
### Deployed Secrets (2026-09-28)
|
||||
- `HandleDeployArtifacts` additionally accepts the command's `Secrets` list (`SecretArtifact`, ciphertext only) and writes the full shipped set to the `deployed_secrets` table via `SiteStorageService.UpsertDeployedSecretsAsync` — upsert every shipped name, soft-remove (`is_removed = 1`) any stored name absent from the set. A `null` `Secrets` field (pre-feature central) touches nothing and answers with no `SecretsApplied`. A current central always ships a list — **empty** when the site references no secrets — so dropping a site's last `secret:` reference removes the secret from both nodes. Storage detail: *Deployed Secrets (Site SQLite / LocalDb)* below.
|
||||
- After writing the table, the active node runs `DeployedSecretsMirror.ReconcileAsync` **synchronously**, before replying — so the deploy response's `SecretsApplied` count and any apply/verify error reflect this deploy's own outcome, not a value left over from the mirror's background schedule. `SecretsApplied` counts only the names *this deploy shipped* that applied and verified cleanly; a mirror failure for an unrelated, already-broken name (a different deploy's leftover, or a stale removal) is logged at Warning and excluded — it neither fails this deploy nor reduces its `SecretsApplied`.
|
||||
- No `DeployedSecretsMirror` registered (secrets deployment not composed on this node) with a non-empty `Secrets` list fails the deploy with `Secret deployment is not enabled on this site node`; with an empty `Secrets` list it still succeeds (`SecretsApplied = 0`).
|
||||
- **Ordering against the DCL.** The shipped secrets are written and mirrored **before** the deploy's data-connection definitions are pushed to the DCL (the definitions are still stored in SQLite at the same point, and the push still goes out if the secrets step fails), so a connection created or reconfigured by the same deploy resolves a newly shipped secret on its first connect. A rotation with **unchanged** connection config does not recreate the connection (the DCL push is hash-guarded) — the new value takes effect on that connection's next (re)connect. No forced reconnect is performed.
|
||||
- See *Deployed Secrets (Site SQLite / LocalDb)* below for the mirror itself.
|
||||
|
||||
### Instance Lifecycle Commands
|
||||
- **Disable**: Stops the Instance Actor and its children. Retains the deployed configuration in SQLite so the instance can be re-enabled without redeployment.
|
||||
- **Enable**: Creates a new Instance Actor from the stored configuration (same as startup).
|
||||
@@ -424,6 +431,27 @@ The Instance Actor owns native-alarm setup alongside its computed Script and Ala
|
||||
|
||||
---
|
||||
|
||||
## Deployed Secrets (Site SQLite / LocalDb)
|
||||
|
||||
Site secrets ride the artifact deploy (design `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`; see that doc's *Implementation notes (as built)* for deviations from the original design). Two tables, two different replication postures:
|
||||
|
||||
- **`deployed_secrets`** — the **11th LocalDb-replicated table** (`SiteLocalDbSetup.ReplicatedTables`, up from ten). Columns: `name` (PK), `row_json` (the shipped `SecretArtifact` — ciphertext, nonce, tag, wrapped DEK, `KekId`, revision; never plaintext), `revision`, `updated_utc`, `is_removed` (soft-remove flag, default 0), `deployment_id`, `applied_at`. No FKs (the standing FK guard — `SiteLocalDbReplicatedTableFkGuardTests` — applies to it like every other replicated table). `UpsertDeployedSecretsAsync` is full-set upsert-then-soft-remove, never a hard `DELETE`, so a removal is itself a replicated row change the standby node can react to.
|
||||
- **`mirrored_secrets`** — a **node-local, unreplicated** ledger (`name` PK, `revision`, `mirrored_at`) recording which names *this specific node* has applied into its own `ISecretStore`. Deliberately not registered with LocalDb: it is a per-node fact (which names this node mirrored), not site-wide state, and replicating it would make one node's mirroring progress masquerade as the other's.
|
||||
|
||||
**`DeployedSecretsMirror`** (`SiteRuntime/Secrets/DeployedSecretsMirror.cs`) is a per-node `BackgroundService` — registered on **every** site node, not a singleton — that reconciles `deployed_secrets` into the node's own local `ISecretStore` (site nodes never run a replication-decorated store; see Component-Host.md → *Secrets*):
|
||||
|
||||
- **Schedule**: runs once immediately at start, then every `SiteRuntime:DeployedSecretsMirrorInterval` (default 15s, validated `> 0`), and synchronously on demand right after an active-node artifact apply (see *Deployed Secrets* under Deployment Manager Singleton above). This 15s interval is the **standby node's own convergence path** — it never receives the active node's synchronous call, only the replicated `deployed_secrets` rows via LocalDb, so it picks them up on its own next tick. **A pair that uses deployed secrets must therefore run LocalDb replication:** on an unreplicated pair only the node that was active at deploy time holds the secrets, and after a failover the new active node has none until the next *Deploy Artifacts* (topology guide → *Secrets: shared store, per-site deploy, and the KEK*).
|
||||
- **Apply is confirm-then-ledger, not blind.** `ApplyReplicatedAsync` is strict last-writer-wins on `(UpdatedUtc, Revision)` (a tie is a no-op) and `DeleteAsync` tombstones rather than hard-deletes — so the mirror always reads the row back after applying and only ledgers it in `mirrored_secrets` when the local ciphertext matches what was shipped. A tombstoned or clock-skewed local row that the ledger already owns is safely re-stamped (`UpdatedUtc`/`Revision` bumped past the local value) and re-applied; a live row with **different** ciphertext that the ledger does **not** already own is left completely untouched — not overwritten, not ledgered, not verified — and reported as an out-of-band conflict (`"exists on this node with a newer local value; the deployed revision was not applied"`), so an operator's genuinely local secret of the same name is never silently destroyed by a later central deploy or removal.
|
||||
- **Ownership, not tombstone-state, decides re-stamping.** Once a name is ledgered in `mirrored_secrets` at any revision, this mirror treats that name as its own going forward — including re-stamping a mirror-owned name that drifted back to the deployed value on the next pass. This holds even when the name is already ledgered at the shipped revision: the mirror reads the local row back every pass, and one that is missing, tombstoned (an operator deleted it locally) or live with different ciphertext (an operator edited it) is healed through the same bumped-stamp re-apply, read-back and cache invalidation before the verify — a local delete is repaired, never misreported as a master-key mismatch. The drift is logged at Warning once per occurrence. Central is authoritative for every name it deploys.
|
||||
- **Unreadable rows are isolated.** A `deployed_secrets` row whose `row_json` cannot be parsed is reported as a failure for that name (in `Failed`/`FailedNames`) instead of aborting the pass; an unreadable **removed** row is still honoured by name.
|
||||
- **Removal** deletes from the local store **only if the name was previously ledgered** (present in `mirrored_secrets`) — an out-of-band local secret sharing a deployed name that was never mirrored is never touched by a central-side removal.
|
||||
- **Verify.** Every pass trial-resolves every live row through `ISecretResolver` (even one not re-applied this pass, so a later KEK change is still caught) — a decrypt failure names the secret and its `KekId` and is reported as a distinct error (de-spammed: Error the first time a given `(name, revision, KekId)` fails, Debug on identical repeats, Error again after an intervening success).
|
||||
- **Isolation.** One row failing to apply/verify does not block any other row in the same pass.
|
||||
- **Cache invalidation.** A successful apply or delete invalidates the shared `ISecretResolver` cache before verifying, so the verify step (and any script/connection reading the secret) sees the fresh value immediately rather than waiting out the resolver's cache TTL.
|
||||
- **Never logs a secret value** — log lines and the reported `MirrorResult` carry only names, revisions, and `KekId`s.
|
||||
|
||||
---
|
||||
|
||||
## Enriched `AlarmStateChanged` Message
|
||||
|
||||
The `AlarmStateChanged` message published by both Alarm Actors and Native Alarm Actors was extended **additively** (existing consumers keep working with computed defaults):
|
||||
@@ -642,10 +670,11 @@ Per Akka.NET best practices, internal actor communication uses **Tell** (fire-an
|
||||
- **Site Event Logging**: Records script executions, alarm events, deployment events, instance lifecycle events.
|
||||
- **Health Monitoring**: Reports script error rates and alarm evaluation error rates.
|
||||
- **Local SQLite**: Persists deployed configurations, system-wide artifacts (external system definitions, database connection definitions, data connection definitions). Sites also maintain peer SQLite stores for the Store-and-Forward buffer, the site event log, the operation tracking table, and the site-local `AuditLog` (see [Component-AuditLog.md](Component-AuditLog.md)). The `AuditLog` file is purged on the same daily cadence as the others but respects the hard `ForwardState` invariant — rows still `Pending` forward are never purged, regardless of age.
|
||||
- **`ZB.MOM.WW.Secrets.Abstractions`** (`ISecretStore`, `ISecretResolver`, `ISecretCacheInvalidator`): `DeployedSecretsMirror` applies deployed secrets into, and verifies against, the node's local secret store (composed by the Host — see Component-Host.md → *Secrets*). The Site Runtime depends only on the abstractions package, never the concrete store implementation.
|
||||
|
||||
## Interactions
|
||||
|
||||
- **Deployment Manager (central)**: Receives flattened configurations, system-wide artifact updates, and instance lifecycle commands.
|
||||
- **Deployment Manager (central)**: Receives flattened configurations, system-wide artifact updates (including a site's referenced secrets, ciphertext only), and instance lifecycle commands.
|
||||
- **Data Connection Layer**: Bidirectional — receives value updates, sends write-back commands.
|
||||
- **Communication Layer**: Receives commands from central, sends deployment results, serves debug view data.
|
||||
- **Store-and-Forward Engine**: Scripts route cached writes, notifications, and external system calls here.
|
||||
|
||||
@@ -5,4 +5,5 @@ public record ArtifactDeploymentResponse(
|
||||
string SiteId,
|
||||
bool Success,
|
||||
string? ErrorMessage,
|
||||
DateTimeOffset Timestamp);
|
||||
DateTimeOffset Timestamp,
|
||||
int? SecretsApplied = null);
|
||||
|
||||
@@ -8,4 +8,13 @@ public record DeployArtifactsCommand(
|
||||
IReadOnlyList<NotificationListArtifact>? NotificationLists,
|
||||
IReadOnlyList<DataConnectionArtifact>? DataConnections,
|
||||
IReadOnlyList<SmtpConfigurationArtifact>? SmtpConfigurations,
|
||||
DateTimeOffset Timestamp);
|
||||
DateTimeOffset Timestamp,
|
||||
IReadOnlyList<SecretArtifact>? Secrets = null,
|
||||
/// <summary>
|
||||
/// Names of secrets referenced by this site's data connections that could not be
|
||||
/// resolved at central (missing or tombstoned). CENTRAL-ONLY: used solely to fold a
|
||||
/// distinguishable marker into <see cref="DeploymentManager.ArtifactRevisionHashService"/>'s
|
||||
/// hash so a site stays flagged out of date until the reference is fixed. Never mapped
|
||||
/// to the wire (<c>SiteCommandDtoMapper</c> does not project it) and never sent to a site.
|
||||
/// </summary>
|
||||
IReadOnlyList<string>? MissingSecrets = null);
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
|
||||
|
||||
/// <summary>
|
||||
/// One secret shipped to a site with its artifacts: the ZB.MOM.WW.Secrets StoredSecret row
|
||||
/// VERBATIM — ciphertext under the fleet's shared KEK, never plaintext. The site applies it
|
||||
/// with ISecretStore.ApplyReplicatedAsync. Design: docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md §3.3.
|
||||
/// </summary>
|
||||
public sealed record SecretArtifact(
|
||||
string Name,
|
||||
string ContentType,
|
||||
byte[] Ciphertext,
|
||||
byte[] Nonce,
|
||||
byte[] Tag,
|
||||
byte[] WrappedDek,
|
||||
byte[] WrapNonce,
|
||||
byte[] WrapTag,
|
||||
string KekId,
|
||||
long Revision,
|
||||
DateTimeOffset CreatedUtc,
|
||||
DateTimeOffset UpdatedUtc,
|
||||
string? Description = null);
|
||||
@@ -0,0 +1,85 @@
|
||||
using System.Text.Json;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Commons.Types.Secrets;
|
||||
|
||||
/// <summary>One secret a site needs, and which of its data connections reference it.</summary>
|
||||
public sealed record SecretReference(string Name, IReadOnlyList<string> ReferencedBy);
|
||||
|
||||
/// <summary>
|
||||
/// Derives the secrets a site needs from its data-connection configs: every JSON string VALUE of the
|
||||
/// exact form <c>secret:<name></c>, at any depth, in primary or backup configuration. Protocol-agnostic
|
||||
/// so a new adapter adopting <c>secret:</c> refs needs no change here. Bootstrap secrets are excluded
|
||||
/// (design §3.1). Unparseable JSON yields nothing rather than throwing — the deploy gate owns config validity.
|
||||
/// </summary>
|
||||
public static class SecretReferenceScanner
|
||||
{
|
||||
public const string Prefix = "secret:";
|
||||
private static readonly string[] BootstrapPrefixes = ["SB-GRPC-PSK-"];
|
||||
|
||||
/// <summary>
|
||||
/// The single parser for a <c>secret:<name></c> reference value, shared by this scanner
|
||||
/// (what central ships) and every adapter that resolves the reference at connect time (what
|
||||
/// the site looks up), so both sides always agree on the name (final review M3). The value is
|
||||
/// trimmed, the <c>secret:</c> prefix matches ordinal-ignore-case, and the remainder is
|
||||
/// trimmed; an empty remainder is not a reference. The config-expander form
|
||||
/// <c>${secret:…}</c> is a different syntax and never matches.
|
||||
/// </summary>
|
||||
/// <param name="value">A configuration string value.</param>
|
||||
/// <param name="name">The referenced secret name when this returns <see langword="true"/>; otherwise empty.</param>
|
||||
/// <returns><see langword="true"/> when <paramref name="value"/> is a <c>secret:<name></c> reference.</returns>
|
||||
public static bool TryParseSecretReference(string? value, out string name)
|
||||
{
|
||||
name = string.Empty;
|
||||
if (value is null) return false;
|
||||
var trimmed = value.Trim();
|
||||
if (!trimmed.StartsWith(Prefix, StringComparison.OrdinalIgnoreCase)) return false;
|
||||
var candidate = trimmed[Prefix.Length..].Trim();
|
||||
if (candidate.Length == 0) return false;
|
||||
name = candidate;
|
||||
return true;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Scans every connection's primary and backup configuration JSON for <c>secret:<name></c>
|
||||
/// string values and returns the distinct names found, ordinal-sorted, each with the ordinal-sorted,
|
||||
/// de-duplicated list of connection names that referenced it.
|
||||
/// </summary>
|
||||
public static IReadOnlyList<SecretReference> Scan(IEnumerable<DataConnectionArtifact> connections)
|
||||
{
|
||||
var found = new SortedDictionary<string, SortedSet<string>>(StringComparer.Ordinal);
|
||||
foreach (var dc in connections)
|
||||
foreach (var json in new[] { dc.PrimaryConfigurationJson, dc.BackupConfigurationJson })
|
||||
foreach (var name in NamesIn(json))
|
||||
{
|
||||
if (!found.TryGetValue(name, out var by)) found[name] = by = new SortedSet<string>(StringComparer.Ordinal);
|
||||
by.Add(dc.Name);
|
||||
}
|
||||
return found.Select(kv => new SecretReference(kv.Key, kv.Value.ToList())).ToList();
|
||||
}
|
||||
|
||||
private static IEnumerable<string> NamesIn(string? json)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(json)) yield break;
|
||||
JsonDocument doc;
|
||||
try { doc = JsonDocument.Parse(json); } catch (JsonException) { yield break; }
|
||||
using (doc)
|
||||
{
|
||||
var stack = new Stack<JsonElement>(); stack.Push(doc.RootElement);
|
||||
while (stack.Count > 0)
|
||||
{
|
||||
var e = stack.Pop();
|
||||
switch (e.ValueKind)
|
||||
{
|
||||
case JsonValueKind.Object: foreach (var p in e.EnumerateObject()) stack.Push(p.Value); break;
|
||||
case JsonValueKind.Array: foreach (var i in e.EnumerateArray()) stack.Push(i); break;
|
||||
case JsonValueKind.String:
|
||||
if (!TryParseSecretReference(e.GetString(), out var name)) break;
|
||||
if (BootstrapPrefixes.Any(b => name.StartsWith(b, StringComparison.OrdinalIgnoreCase))) break;
|
||||
yield return name;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,3 +1,4 @@
|
||||
using Google.Protobuf;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Protocol;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection;
|
||||
@@ -444,6 +445,13 @@ public static class SiteCommandDtoMapper
|
||||
dto.SmtpConfigurations = list;
|
||||
}
|
||||
|
||||
if (command.Secrets is not null)
|
||||
{
|
||||
var list = new SecretArtifactListDto();
|
||||
list.Items.AddRange(command.Secrets.Select(ToProto));
|
||||
dto.Secrets = list;
|
||||
}
|
||||
|
||||
return dto;
|
||||
}
|
||||
|
||||
@@ -462,7 +470,56 @@ public static class SiteCommandDtoMapper
|
||||
dto.NotificationLists is null ? null : dto.NotificationLists.Items.Select(FromProto).ToList(),
|
||||
dto.DataConnections is null ? null : dto.DataConnections.Items.Select(FromProto).ToList(),
|
||||
dto.SmtpConfigurations is null ? null : dto.SmtpConfigurations.Items.Select(FromProto).ToList(),
|
||||
Dto(dto.Timestamp));
|
||||
Dto(dto.Timestamp),
|
||||
dto.Secrets is null ? null : dto.Secrets.Items.Select(FromProto).ToList());
|
||||
}
|
||||
|
||||
/// <summary>Projects a <see cref="SecretArtifact"/> onto the wire.</summary>
|
||||
/// <param name="artifact">The artifact to project.</param>
|
||||
/// <returns>The wire message.</returns>
|
||||
public static SecretArtifactDto ToProto(SecretArtifact artifact)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(artifact);
|
||||
|
||||
return new SecretArtifactDto
|
||||
{
|
||||
Name = artifact.Name,
|
||||
ContentType = artifact.ContentType,
|
||||
Ciphertext = ByteString.CopyFrom(artifact.Ciphertext),
|
||||
Nonce = ByteString.CopyFrom(artifact.Nonce),
|
||||
Tag = ByteString.CopyFrom(artifact.Tag),
|
||||
WrappedDek = ByteString.CopyFrom(artifact.WrappedDek),
|
||||
WrapNonce = ByteString.CopyFrom(artifact.WrapNonce),
|
||||
WrapTag = ByteString.CopyFrom(artifact.WrapTag),
|
||||
KekId = artifact.KekId,
|
||||
Revision = artifact.Revision,
|
||||
CreatedUtc = Ts(artifact.CreatedUtc),
|
||||
UpdatedUtc = Ts(artifact.UpdatedUtc),
|
||||
Description = artifact.Description
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>Rehydrates a <see cref="SecretArtifact"/> from the wire.</summary>
|
||||
/// <param name="dto">The wire message to decode.</param>
|
||||
/// <returns>The decoded artifact.</returns>
|
||||
public static SecretArtifact FromProto(SecretArtifactDto dto)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(dto);
|
||||
|
||||
return new SecretArtifact(
|
||||
dto.Name,
|
||||
dto.ContentType,
|
||||
dto.Ciphertext.ToByteArray(),
|
||||
dto.Nonce.ToByteArray(),
|
||||
dto.Tag.ToByteArray(),
|
||||
dto.WrappedDek.ToByteArray(),
|
||||
dto.WrapNonce.ToByteArray(),
|
||||
dto.WrapTag.ToByteArray(),
|
||||
dto.KekId,
|
||||
dto.Revision,
|
||||
Dto(dto.CreatedUtc),
|
||||
Dto(dto.UpdatedUtc),
|
||||
dto.Description);
|
||||
}
|
||||
|
||||
/// <summary>Projects a <see cref="SharedScriptArtifact"/> onto the wire.</summary>
|
||||
@@ -756,7 +813,7 @@ public static class SiteCommandDtoMapper
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(response);
|
||||
|
||||
return new ArtifactDeploymentResponseDto
|
||||
var dto = new ArtifactDeploymentResponseDto
|
||||
{
|
||||
DeploymentId = response.DeploymentId,
|
||||
SiteId = response.SiteId,
|
||||
@@ -764,6 +821,13 @@ public static class SiteCommandDtoMapper
|
||||
ErrorMessage = OrEmpty(response.ErrorMessage),
|
||||
Timestamp = Ts(response.Timestamp)
|
||||
};
|
||||
|
||||
if (response.SecretsApplied.HasValue)
|
||||
{
|
||||
dto.SecretsApplied = response.SecretsApplied.Value;
|
||||
}
|
||||
|
||||
return dto;
|
||||
}
|
||||
|
||||
/// <summary>Rehydrates an <see cref="ArtifactDeploymentResponse"/> from the wire.</summary>
|
||||
@@ -774,7 +838,8 @@ public static class SiteCommandDtoMapper
|
||||
ArgumentNullException.ThrowIfNull(dto);
|
||||
|
||||
return new ArtifactDeploymentResponse(
|
||||
dto.DeploymentId, dto.SiteId, dto.Success, NullIfEmpty(dto.ErrorMessage), Dto(dto.Timestamp));
|
||||
dto.DeploymentId, dto.SiteId, dto.Success, NullIfEmpty(dto.ErrorMessage), Dto(dto.Timestamp),
|
||||
dto.SecretsApplied);
|
||||
}
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────
|
||||
|
||||
@@ -270,6 +270,25 @@ message SmtpConfigurationArtifactDto {
|
||||
// own message makes the null/empty distinction a message-presence question,
|
||||
// which proto3 does model — the same silent-data-loss class the transport
|
||||
// round-trip guard caught in PLAN-05 T8.
|
||||
// One StoredSecret row, ciphertext verbatim under the fleet's shared KEK — never
|
||||
// plaintext on the wire. Site secrets design §3.3.
|
||||
message SecretArtifactDto {
|
||||
string name = 1;
|
||||
string content_type = 2;
|
||||
bytes ciphertext = 3;
|
||||
bytes nonce = 4;
|
||||
bytes tag = 5;
|
||||
bytes wrapped_dek = 6;
|
||||
bytes wrap_nonce = 7;
|
||||
bytes wrap_tag = 8;
|
||||
string kek_id = 9;
|
||||
int64 revision = 10;
|
||||
google.protobuf.Timestamp created_utc = 11;
|
||||
google.protobuf.Timestamp updated_utc = 12;
|
||||
google.protobuf.StringValue description = 13; // absent => null
|
||||
}
|
||||
message SecretArtifactListDto { repeated SecretArtifactDto items = 1; }
|
||||
|
||||
message SharedScriptArtifactListDto { repeated SharedScriptArtifactDto items = 1; }
|
||||
message ExternalSystemArtifactListDto { repeated ExternalSystemArtifactDto items = 1; }
|
||||
message DatabaseConnectionArtifactListDto { repeated DatabaseConnectionArtifactDto items = 1; }
|
||||
@@ -286,6 +305,7 @@ message DeployArtifactsCommandDto {
|
||||
DataConnectionArtifactListDto data_connections = 6; // absent => null
|
||||
SmtpConfigurationArtifactListDto smtp_configurations = 7; // absent => null
|
||||
google.protobuf.Timestamp timestamp = 8;
|
||||
SecretArtifactListDto secrets = 9; // absent => null
|
||||
}
|
||||
|
||||
message DeploymentStatusResponseDto {
|
||||
@@ -319,6 +339,7 @@ message ArtifactDeploymentResponseDto {
|
||||
bool success = 3;
|
||||
string error_message = 4; // empty string represents null
|
||||
google.protobuf.Timestamp timestamp = 5;
|
||||
google.protobuf.Int32Value secrets_applied = 6; // absent => null
|
||||
}
|
||||
|
||||
message LifecycleRequest {
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -3,6 +3,7 @@ using Microsoft.Extensions.Logging;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Protocol;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Serialization;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Secrets;
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Adapters;
|
||||
@@ -98,20 +99,27 @@ public class MxGatewayDataConnection
|
||||
// so a fail-closed throw here leaves adapter state untouched (no half-open client,
|
||||
// previous connection — if any — still intact for a later retry). A literal key
|
||||
// (no `secret:` prefix) passes through unchanged for back-compat.
|
||||
// The reference is parsed by the SAME parser central's SecretReferenceScanner uses to
|
||||
// decide which secret to ship (trimmed, case-insensitive prefix), so a padded or
|
||||
// case-variant reference resolves exactly the name that was deployed (final review M3).
|
||||
var apiKey = cfg.ApiKey;
|
||||
if (!string.IsNullOrEmpty(apiKey) &&
|
||||
apiKey.StartsWith("secret:", StringComparison.OrdinalIgnoreCase))
|
||||
if (SecretReferenceScanner.TryParseSecretReference(apiKey, out var secretName))
|
||||
{
|
||||
if (_secretResolver is null)
|
||||
{
|
||||
throw new InvalidOperationException(
|
||||
"MxGateway ApiKey uses a 'secret:' reference but no ISecretResolver is available to resolve it.");
|
||||
}
|
||||
var secretName = apiKey["secret:".Length..];
|
||||
apiKey = await _secretResolver.GetAsync(new SecretName(secretName), cancellationToken)
|
||||
?? throw new InvalidOperationException(
|
||||
$"MxGateway ApiKey secret '{secretName}' could not be resolved (not found or tombstoned).");
|
||||
}
|
||||
else if (!string.IsNullOrEmpty(apiKey) &&
|
||||
apiKey.TrimStart().StartsWith(SecretReferenceScanner.Prefix, StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
// `secret:` with no name: fail closed rather than sending the literal text as a key.
|
||||
throw new InvalidOperationException("MxGateway ApiKey is a 'secret:' reference with no secret name.");
|
||||
}
|
||||
|
||||
// Reconnect reuses this adapter: cancel the previous event loop and dispose
|
||||
// the previous client so a flapping gateway cannot accumulate orphaned
|
||||
|
||||
@@ -7,6 +7,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Secrets;
|
||||
using ZB.MOM.WW.ScadaBridge.Communication;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.DeploymentManager;
|
||||
@@ -40,6 +41,7 @@ public class ArtifactDeploymentService
|
||||
private readonly DeploymentManagerOptions _options;
|
||||
private readonly ILogger<ArtifactDeploymentService> _logger;
|
||||
private readonly ArtifactRevisionHashService _hashService;
|
||||
private readonly SiteSecretsCollector _secretsCollector;
|
||||
|
||||
/// <summary>
|
||||
/// Initializes a new instance of the ArtifactDeploymentService.
|
||||
@@ -60,8 +62,13 @@ public class ArtifactDeploymentService
|
||||
/// <param name="logger">Logger instance.</param>
|
||||
/// <param name="hashService">
|
||||
/// Computes the deterministic revision hash of a per-site artifact set, stamped on the
|
||||
/// <see cref="Site"/> row after a successful deploy. Last on the signature so the
|
||||
/// addition stays positionally additive for existing call sites.
|
||||
/// <see cref="Site"/> row after a successful deploy. Second-to-last on the signature so
|
||||
/// the addition stayed positionally additive for existing call sites at the time it was
|
||||
/// introduced.
|
||||
/// </param>
|
||||
/// <param name="secretsCollector">
|
||||
/// Resolves a site's referenced secrets (ciphertext rows) against the central secret
|
||||
/// store (design §3.1/§3.3). Last on the signature for the same reason.
|
||||
/// </param>
|
||||
public ArtifactDeploymentService(
|
||||
ISiteRepository siteRepo,
|
||||
@@ -73,7 +80,8 @@ public class ArtifactDeploymentService
|
||||
IAuditService auditService,
|
||||
IOptions<DeploymentManagerOptions> options,
|
||||
ILogger<ArtifactDeploymentService> logger,
|
||||
ArtifactRevisionHashService hashService)
|
||||
ArtifactRevisionHashService hashService,
|
||||
SiteSecretsCollector secretsCollector)
|
||||
{
|
||||
_siteRepo = siteRepo;
|
||||
_deploymentRepo = deploymentRepo;
|
||||
@@ -87,6 +95,7 @@ public class ArtifactDeploymentService
|
||||
_options = options.Value;
|
||||
_logger = logger;
|
||||
_hashService = hashService;
|
||||
_secretsCollector = secretsCollector;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -101,7 +110,12 @@ public class ArtifactDeploymentService
|
||||
/// UI summary, and persisted record correlate. When <c>null</c> a fresh id is minted (used by
|
||||
/// single-site retries).
|
||||
/// </param>
|
||||
/// <returns>A deployment artifacts command for the site.</returns>
|
||||
/// <returns>
|
||||
/// The deployment artifacts command for the site, and any secret references its data
|
||||
/// connections make that could not be resolved at central (absent or tombstoned). A
|
||||
/// non-empty <c>Missing</c> list means the command must NOT be sent to the site —
|
||||
/// see <see cref="DeployCoreAsync"/>/<see cref="RetryForSiteAsync"/>.
|
||||
/// </returns>
|
||||
/// <remarks>
|
||||
/// This convenience overload runs the global artifact queries
|
||||
/// for a single site (used by <see cref="RetryForSiteAsync"/>). The multi-site
|
||||
@@ -109,7 +123,7 @@ public class ArtifactDeploymentService
|
||||
/// per-site loop and calls the prefetched-globals overload to avoid the N+1
|
||||
/// re-query of every system-wide artifact set per site.
|
||||
/// </remarks>
|
||||
public async Task<DeployArtifactsCommand> BuildDeployArtifactsCommandAsync(
|
||||
public async Task<(DeployArtifactsCommand Command, IReadOnlyList<SecretReference> Missing)> BuildDeployArtifactsCommandAsync(
|
||||
int siteId,
|
||||
CancellationToken cancellationToken = default,
|
||||
string? deploymentId = null)
|
||||
@@ -121,7 +135,8 @@ public class ArtifactDeploymentService
|
||||
/// <summary>
|
||||
/// Builds a per-site <see cref="DeployArtifactsCommand"/> using a previously-fetched
|
||||
/// snapshot of the global artifact sets (shared scripts, external systems + methods,
|
||||
/// DB connections). Only the per-site data-connection query runs here.
|
||||
/// DB connections). Only the per-site data-connection query — and the per-site secret
|
||||
/// resolution it drives — runs here.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Separating the global fetch from the per-site build lets
|
||||
@@ -134,8 +149,15 @@ public class ArtifactDeploymentService
|
||||
/// delivery is central-only and no notification artifact or SMTP credential is
|
||||
/// ever distributed to a site. The fields remain on the contract only for
|
||||
/// additive compatibility.
|
||||
///
|
||||
/// The command's <c>Secrets</c>/<c>MissingSecrets</c> fields (design §3.1/§3.3) are
|
||||
/// populated from <see cref="SiteSecretsCollector.CollectAsync"/> regardless of whether
|
||||
/// any reference is missing — <c>MissingSecrets</c> is CENTRAL-ONLY (never mapped to the
|
||||
/// wire) and exists solely so <see cref="ArtifactRevisionHashService"/> can fold a
|
||||
/// distinguishable marker into the hash even for a command the caller ultimately refuses
|
||||
/// to send (see the returned <c>Missing</c> tuple element).
|
||||
/// </remarks>
|
||||
private async Task<DeployArtifactsCommand> BuildDeployArtifactsCommandAsync(
|
||||
private async Task<(DeployArtifactsCommand Command, IReadOnlyList<SecretReference> Missing)> BuildDeployArtifactsCommandAsync(
|
||||
int siteId,
|
||||
GlobalArtifactSnapshot globals,
|
||||
CancellationToken cancellationToken,
|
||||
@@ -147,7 +169,9 @@ public class ArtifactDeploymentService
|
||||
var dataConnectionArtifacts = dataConnections.Select(dc =>
|
||||
new DataConnectionArtifact(dc.Name, dc.Protocol, dc.PrimaryConfiguration, dc.BackupConfiguration, dc.FailoverRetryCount)).ToList();
|
||||
|
||||
return new DeployArtifactsCommand(
|
||||
var collected = await _secretsCollector.CollectAsync(dataConnectionArtifacts, cancellationToken);
|
||||
|
||||
var command = new DeployArtifactsCommand(
|
||||
deploymentId ?? Guid.NewGuid().ToString("N"),
|
||||
globals.SharedScripts,
|
||||
globals.ExternalSystems,
|
||||
@@ -157,9 +181,111 @@ public class ArtifactDeploymentService
|
||||
dataConnectionArtifacts,
|
||||
// SMTP config (incl. credentials) is central-only — never shipped to sites.
|
||||
SmtpConfigurations: null,
|
||||
DateTimeOffset.UtcNow);
|
||||
DateTimeOffset.UtcNow,
|
||||
// ALWAYS a list — never null — even when the site references no secrets. The site
|
||||
// applies Secrets as the FULL set (every deployed_secrets row absent from it is
|
||||
// marked removed and the mirror deletes it), and a null Secrets means "central did
|
||||
// not ship secrets at all" (the pre-feature/old-central contract), which touches
|
||||
// nothing. Shipping null here when the LAST reference is dropped therefore left the
|
||||
// secret live on both site nodes forever (Task 13 live gate, check 5). An empty
|
||||
// list costs an old site nothing (it ignores proto field 9), and the hash is
|
||||
// unaffected: the hash service folds an empty and a null list identically.
|
||||
Secrets: collected.Secrets,
|
||||
MissingSecrets: collected.Missing.Count > 0 ? collected.Missing.Select(m => m.Name).ToList() : null);
|
||||
|
||||
return (command, collected.Missing);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Task 7 (design §3.4): reconciles a site's raw <see cref="ArtifactDeploymentResponse"/>
|
||||
/// against what the deployed <see cref="DeployArtifactsCommand"/> actually shipped, so
|
||||
/// central never stamps a site *Current* on the strength of a reply that only proves the
|
||||
/// artifact set was RECEIVED, not that every shipped secret was APPLIED.
|
||||
///
|
||||
/// A site that predates the secrets feature answers with <c>SecretsApplied = null</c> even
|
||||
/// on an otherwise-successful apply — silently ignoring the new proto field rather than
|
||||
/// failing loudly — so that case must be caught here rather than trusted as success. A site
|
||||
/// that applied some but not all shipped secrets (a partial local failure) reports a count
|
||||
/// that undershoots what was sent. Both are downgraded to a failure; the site's previous
|
||||
/// stamp (or lack of one) is left untouched because stamping already keys off
|
||||
/// <see cref="SiteArtifactResult.Success"/>.
|
||||
///
|
||||
/// Only fires when the command actually shipped secrets — a site with nothing to apply
|
||||
/// truthfully has nothing to confirm, so its <c>SecretsApplied</c> stays uninterpreted
|
||||
/// (old-site behaviour is unchanged when no secrets are in play, per the acceptance
|
||||
/// criteria).
|
||||
/// </summary>
|
||||
private static SiteArtifactResult ReconcileSecretsOutcome(
|
||||
DeployArtifactsCommand cmd, ArtifactDeploymentResponse r, string siteIdentifier, string siteName)
|
||||
{
|
||||
var shipped = cmd.Secrets?.Count ?? 0;
|
||||
if (r.Success && shipped > 0 && r.SecretsApplied is null)
|
||||
return new(siteIdentifier, siteName, false,
|
||||
"Site build predates secret deployment — upgrade the site before deploying secrets", null);
|
||||
if (r.Success && shipped > 0 && r.SecretsApplied != shipped)
|
||||
return new(siteIdentifier, siteName, false,
|
||||
$"Site applied {r.SecretsApplied} of {shipped} secrets", r.SecretsApplied);
|
||||
return new(siteIdentifier, siteName, r.Success, r.ErrorMessage, shipped > 0 ? r.SecretsApplied : null);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Best-effort <c>DeploySiteSecrets</c> audit row (task 7): one row per site whose command
|
||||
/// carried, or would have carried, at least one secret — a resolved
|
||||
/// <see cref="DeployArtifactsCommand.Secrets"/> entry, an unresolved reference in
|
||||
/// <paramref name="missing"/> (the fail-closed case, where nothing was sent), or both. A
|
||||
/// site with no secret references at all gets no row.
|
||||
///
|
||||
/// Per the audit-is-best-effort rule (an audit failure never changes the deploy outcome)
|
||||
/// this is fire-and-forget-shaped only in intent, not mechanism — like every other audit
|
||||
/// call on this path it is still awaited (never on a script path; this is a management
|
||||
/// operation), but a thrown/faulted write here must not — and structurally cannot, since
|
||||
/// the caller already decided <see cref="SiteArtifactResult.Success"/> beforehand — change
|
||||
/// whether the site gets stamped.
|
||||
///
|
||||
/// The <c>afterState</c> payload is built ONLY from names/revisions/missing names/success/
|
||||
/// error — never <see cref="SecretArtifact.Ciphertext"/>, <c>Nonce</c>, <c>Tag</c>,
|
||||
/// <c>WrappedDek</c>, or any other material (global constraint: never plaintext or material
|
||||
/// in logs or audit rows).
|
||||
/// </summary>
|
||||
private Task LogSecretsAuditAsync(
|
||||
string user,
|
||||
DeployArtifactsCommand command,
|
||||
IReadOnlyList<SecretReference> missing,
|
||||
string siteIdentifier,
|
||||
string siteName,
|
||||
bool success,
|
||||
string? error,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var hasSecretRefs = (command.Secrets?.Count ?? 0) > 0 || missing.Count > 0;
|
||||
if (!hasSecretRefs)
|
||||
return Task.CompletedTask;
|
||||
|
||||
return _auditService.LogAsync(
|
||||
user, "DeploySiteSecrets", "SiteSecrets", siteIdentifier, siteName,
|
||||
new
|
||||
{
|
||||
deploymentId = command.DeploymentId,
|
||||
secrets = (command.Secrets ?? [])
|
||||
.Select(s => new { name = s.Name, revision = s.Revision })
|
||||
.ToList(),
|
||||
missing = missing.Select(m => m.Name).ToList(),
|
||||
success,
|
||||
error,
|
||||
},
|
||||
cancellationToken);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Formats the fail-closed message for one or more secret references a site's data
|
||||
/// connections make that could not be resolved at central (design §3.3). Never includes
|
||||
/// secret material — only names and the referencing connection names.
|
||||
/// </summary>
|
||||
private static string BuildMissingSecretsMessage(IReadOnlyList<SecretReference> missing) =>
|
||||
string.Join("; ", missing.Select(m =>
|
||||
$"Secret '{m.Name}' referenced by data connection(s) {string.Join(", ", m.ReferencedBy)} " +
|
||||
"is not in the central secret store"));
|
||||
|
||||
/// <summary>
|
||||
/// Fetches the system-wide artifact sets that are identical across every site —
|
||||
/// shared scripts, external systems (with their methods serialized in), and
|
||||
@@ -297,7 +423,11 @@ public class ArtifactDeploymentService
|
||||
// Every per-site command carries the SAME logical
|
||||
// deploymentId, so the per-site commands, audit log, persisted record,
|
||||
// and UI summary all reference one id instead of N+1 unrelated GUIDs.
|
||||
var siteCommands = new Dictionary<int, DeployArtifactsCommand>();
|
||||
//
|
||||
// Each entry also carries the site's unresolved secret references (design §3.3):
|
||||
// a non-empty Missing list means this site's command must never reach
|
||||
// _communicationService — see the fan-out loop below.
|
||||
var siteCommands = new Dictionary<int, (DeployArtifactsCommand Command, IReadOnlyList<SecretReference> Missing)>();
|
||||
foreach (var site in sites)
|
||||
{
|
||||
siteCommands[site.Id] = await BuildDeployArtifactsCommandAsync(
|
||||
@@ -327,7 +457,28 @@ public class ArtifactDeploymentService
|
||||
using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
|
||||
cts.CancelAfter(_options.ArtifactDeploymentTimeoutPerSite);
|
||||
|
||||
var command = siteCommands[site.Id];
|
||||
var (command, missing) = siteCommands[site.Id];
|
||||
|
||||
// Fail this site closed BEFORE anything is sent (design §3.3): a referenced
|
||||
// secret that is absent or tombstoned at central must not let the rest of the
|
||||
// artifact set ship, or the site would read *Current* while a connection
|
||||
// cannot resolve its key. Other sites in the batch are unaffected.
|
||||
if (missing.Count > 0)
|
||||
{
|
||||
var missingMessage = BuildMissingSecretsMessage(missing);
|
||||
_logger.LogError(
|
||||
"Cannot deploy artifacts to site {SiteId} ({SiteName}): {Error}",
|
||||
site.SiteIdentifier, site.Name, missingMessage);
|
||||
return new SiteArtifactResult(site.SiteIdentifier, site.Name, false, missingMessage);
|
||||
}
|
||||
|
||||
if (command.Secrets is { Count: > 0 })
|
||||
{
|
||||
_logger.LogInformation(
|
||||
"Deploying {Count} secret(s) to site {SiteId}: {Secrets}",
|
||||
command.Secrets.Count, site.SiteIdentifier,
|
||||
string.Join(", ", command.Secrets.Select(s => $"{s.Name}@r{s.Revision}")));
|
||||
}
|
||||
|
||||
_logger.LogInformation(
|
||||
"Deploying artifacts to site {SiteId} ({SiteName}), deploymentId={DeploymentId}",
|
||||
@@ -336,8 +487,10 @@ public class ArtifactDeploymentService
|
||||
var response = await _communicationService.DeployArtifactsAsync(
|
||||
site.SiteIdentifier, command, cts.Token);
|
||||
|
||||
return new SiteArtifactResult(
|
||||
site.SiteIdentifier, site.Name, response.Success, response.ErrorMessage);
|
||||
// Task 7 (design §3.4): a reply that only proves receipt is not enough to
|
||||
// stamp *Current* — reconcile it against what was actually shipped before
|
||||
// this result feeds the stamp decision below.
|
||||
return ReconcileSecretsOutcome(command, response, site.SiteIdentifier, site.Name);
|
||||
}
|
||||
catch (Exception ex) when (ex is TimeoutException or OperationCanceledException or TaskCanceledException)
|
||||
{
|
||||
@@ -366,6 +519,22 @@ public class ArtifactDeploymentService
|
||||
perSiteResults[result.SiteId] = result;
|
||||
}
|
||||
|
||||
// Task 7 (design §3.4): one best-effort DeploySiteSecrets audit row per site whose
|
||||
// command carried or would have carried secrets — covers the reconciled
|
||||
// success/failure path AND the fail-closed (missing secret, nothing sent) path, since
|
||||
// both already flow into perSiteResults above by the time this loop runs. Driven off
|
||||
// siteCommands/perSiteResults rather than inlined into the per-site task above so this
|
||||
// is a single serial pass (no concurrent audit writes) that runs — per the ordering
|
||||
// rule — BEFORE the best-effort stamp below.
|
||||
foreach (var site in sites)
|
||||
{
|
||||
var (command, missing) = siteCommands[site.Id];
|
||||
var result = perSiteResults[site.SiteIdentifier];
|
||||
await LogSecretsAuditAsync(
|
||||
user, command, missing, site.SiteIdentifier, site.Name,
|
||||
result.Success, result.ErrorMessage, cancellationToken);
|
||||
}
|
||||
|
||||
// Persist the system artifact deployment record.
|
||||
// SystemArtifactDeploymentRecord has no dedicated
|
||||
// DeploymentId column (adding one is a Commons/ConfigurationDatabase
|
||||
@@ -405,7 +574,7 @@ public class ArtifactDeploymentService
|
||||
if (successfulSites.Count > 0)
|
||||
{
|
||||
await StampArtifactRevisionAsync(
|
||||
successfulSites.Select(site => (site, siteCommands[site.Id])).ToList(),
|
||||
successfulSites.Select(site => (site, siteCommands[site.Id].Command)).ToList(),
|
||||
cancellationToken);
|
||||
}
|
||||
|
||||
@@ -442,13 +611,45 @@ public class ArtifactDeploymentService
|
||||
using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
|
||||
cts.CancelAfter(_options.ArtifactDeploymentTimeoutPerSite);
|
||||
|
||||
var command = await BuildDeployArtifactsCommandAsync(siteDbId, cts.Token);
|
||||
var (command, missing) = await BuildDeployArtifactsCommandAsync(siteDbId, cts.Token);
|
||||
|
||||
// Same fail-closed rule as the sweep path (design §3.3): a retry must not ship a
|
||||
// site an artifact set whose secret references still don't resolve.
|
||||
if (missing.Count > 0)
|
||||
{
|
||||
var missingMessage = BuildMissingSecretsMessage(missing);
|
||||
_logger.LogError(
|
||||
"Cannot deploy artifacts to site {SiteId}: {Error}", siteIdentifier, missingMessage);
|
||||
// Task 7: the fail-closed case still gets its DeploySiteSecrets row — nothing
|
||||
// was sent, but the command carried (or would have carried) secrets.
|
||||
await LogSecretsAuditAsync(
|
||||
user, command, missing, siteIdentifier, siteIdentifier, false, missingMessage,
|
||||
cancellationToken);
|
||||
return Result<SiteArtifactResult>.Failure(missingMessage);
|
||||
}
|
||||
|
||||
if (command.Secrets is { Count: > 0 })
|
||||
{
|
||||
_logger.LogInformation(
|
||||
"Deploying {Count} secret(s) to site {SiteId}: {Secrets}",
|
||||
command.Secrets.Count, siteIdentifier,
|
||||
string.Join(", ", command.Secrets.Select(s => $"{s.Name}@r{s.Revision}")));
|
||||
}
|
||||
|
||||
var response = await _communicationService.DeployArtifactsAsync(siteIdentifier, command, cts.Token);
|
||||
|
||||
var result = new SiteArtifactResult(siteIdentifier, siteIdentifier, response.Success, response.ErrorMessage);
|
||||
// Task 7 (design §3.4): reconcile the raw wire response against what was
|
||||
// actually shipped before it feeds the audit row, the stamp decision, or the
|
||||
// returned Result — an old site's/partial site's reply must not read as success.
|
||||
var result = ReconcileSecretsOutcome(command, response, siteIdentifier, siteIdentifier);
|
||||
|
||||
await _auditService.LogAsync(user, "RetryArtifactDeployment", "SystemArtifact",
|
||||
command.DeploymentId, siteIdentifier, new { response.Success }, cancellationToken);
|
||||
command.DeploymentId, siteIdentifier, new { result.Success }, cancellationToken);
|
||||
|
||||
// Ordering rule: audit before the best-effort stamp.
|
||||
await LogSecretsAuditAsync(
|
||||
user, command, missing, siteIdentifier, siteIdentifier,
|
||||
result.Success, result.ErrorMessage, cancellationToken);
|
||||
|
||||
// Same stamp-on-success rule as the sweep path: a retry that lands is a
|
||||
// successful artifact deploy and must move the site's revision stamp; a
|
||||
@@ -459,14 +660,14 @@ public class ArtifactDeploymentService
|
||||
// an artifact deploy that actually succeeded, so the stamp must (and does) keep
|
||||
// its own try/catch inside StampArtifactRevisionAsync rather than lean on this
|
||||
// one.
|
||||
if (response.Success)
|
||||
if (result.Success)
|
||||
{
|
||||
await StampRetriedSiteAsync(siteDbId, command, cancellationToken);
|
||||
}
|
||||
|
||||
return response.Success
|
||||
return result.Success
|
||||
? Result<SiteArtifactResult>.Success(result)
|
||||
: Result<SiteArtifactResult>.Failure(response.ErrorMessage ?? "Retry failed.");
|
||||
: Result<SiteArtifactResult>.Failure(result.ErrorMessage ?? "Retry failed.");
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
@@ -512,7 +713,11 @@ public class ArtifactDeploymentService
|
||||
var globals = await FetchGlobalArtifactsAsync(cancellationToken);
|
||||
foreach (var site in sites)
|
||||
{
|
||||
var command = await BuildDeployArtifactsCommandAsync(
|
||||
// The Missing tuple element is deliberately unused here: ComputeHash reads
|
||||
// MissingSecrets straight off the command (design §3.4), so an unresolved
|
||||
// reference still folds a distinguishable marker into the hash — the site reads
|
||||
// *Out of date* instead of this call failing the whole batch.
|
||||
var (command, _) = await BuildDeployArtifactsCommandAsync(
|
||||
site.Id, globals, cancellationToken, deploymentId: "current");
|
||||
hashes[site.Id] = _hashService.ComputeHash(command);
|
||||
}
|
||||
@@ -587,11 +792,20 @@ public class ArtifactDeploymentService
|
||||
/// <summary>
|
||||
/// Per-site result for artifact deployment.
|
||||
/// </summary>
|
||||
/// <param name="SecretsDeployed">
|
||||
/// How many secrets the site confirmed applying, when the deployed command shipped at
|
||||
/// least one (task 7, design §3.4). <c>null</c> when the site shipped no secrets at all
|
||||
/// — deliberately the same value an old site's unset <see cref="ArtifactDeploymentResponse.SecretsApplied"/>
|
||||
/// produces, so the two "no secrets involved" cases (nothing shipped vs. an old site that
|
||||
/// doesn't know about the field) are indistinguishable, which is correct: neither one is a
|
||||
/// secrets-gating concern.
|
||||
/// </param>
|
||||
public record SiteArtifactResult(
|
||||
string SiteId,
|
||||
string SiteName,
|
||||
bool Success,
|
||||
string? ErrorMessage);
|
||||
string? ErrorMessage,
|
||||
int? SecretsDeployed = null);
|
||||
|
||||
/// <summary>
|
||||
/// Summary of system-wide artifact deployment with per-site results.
|
||||
|
||||
@@ -34,6 +34,16 @@ namespace ZB.MOM.WW.ScadaBridge.DeploymentManager;
|
||||
/// in would only add a permanently-null branch).
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// INCLUDED: the command's <c>Secrets</c> — projected as a content digest (SHA-256 of
|
||||
/// ciphertext + nonce + tag + wrapped-DEK + wrap-nonce + wrap-tag) alongside the name,
|
||||
/// revision, and KekId, so plaintext never enters the hash input and re-wrapping under
|
||||
/// a new KEK with unchanged content still moves the hash. <c>MissingSecrets</c> —
|
||||
/// central-only names of secret references that failed to resolve — is folded in as
|
||||
/// distinguishable marker entries (<c>Missing = true</c>, no digest) so a site stays
|
||||
/// flagged out of date while a reference is broken, and a resolved secret hashes
|
||||
/// differently from the same name unresolved.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// A no-op re-save of a data connection does NOT change the hash. Connection
|
||||
/// configuration is encrypted at rest through an EF value converter, so the
|
||||
/// ciphertext changes on every save, but the entity — and hence
|
||||
@@ -111,6 +121,7 @@ public sealed class ArtifactRevisionHashService
|
||||
})
|
||||
.ToList()
|
||||
: null,
|
||||
Secrets = BuildHashableSecrets(command),
|
||||
SharedScripts = command.SharedScripts is { Count: > 0 }
|
||||
? command.SharedScripts
|
||||
.OrderBy(s => s.Name, StringComparer.Ordinal)
|
||||
@@ -130,6 +141,42 @@ public sealed class ArtifactRevisionHashService
|
||||
return $"sha256:{Convert.ToHexStringLower(hashBytes)}";
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Projects the command's <c>Secrets</c> and <c>MissingSecrets</c> into a single
|
||||
/// name-sorted list of <see cref="HashableSecret"/> entries. A present secret carries a
|
||||
/// content digest over its ciphertext material (never the plaintext, which this service
|
||||
/// never sees) plus its <c>KekId</c> and <c>Revision</c>; a missing reference carries only
|
||||
/// its name and <c>Missing = true</c>, so the same name present vs. missing always hashes
|
||||
/// differently. Returns null when there is nothing to hash, matching every other
|
||||
/// null-when-empty artifact list here.
|
||||
/// </summary>
|
||||
private static List<HashableSecret>? BuildHashableSecrets(DeployArtifactsCommand command)
|
||||
{
|
||||
var list = new List<HashableSecret>();
|
||||
|
||||
foreach (var s in command.Secrets ?? [])
|
||||
{
|
||||
list.Add(new HashableSecret
|
||||
{
|
||||
ContentDigest = Convert.ToHexStringLower(SHA256.HashData(
|
||||
[.. s.Ciphertext, .. s.Nonce, .. s.Tag, .. s.WrappedDek, .. s.WrapNonce, .. s.WrapTag])),
|
||||
KekId = s.KekId,
|
||||
Missing = null,
|
||||
Name = s.Name,
|
||||
Revision = s.Revision
|
||||
});
|
||||
}
|
||||
|
||||
foreach (var m in command.MissingSecrets ?? [])
|
||||
{
|
||||
list.Add(new HashableSecret { Missing = true, Name = m });
|
||||
}
|
||||
|
||||
return list.Count == 0
|
||||
? null
|
||||
: list.OrderBy(x => x.Name, StringComparer.Ordinal).ThenBy(x => x.Missing == true).ToList();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Re-serializes an external system's JSON-encoded method list with its methods sorted
|
||||
/// by name (ordinal), so the repository's row order cannot leak into the hash. Anything
|
||||
@@ -211,6 +258,11 @@ public sealed class ArtifactRevisionHashService
|
||||
/// </summary>
|
||||
public List<HashableExternalSystem>? ExternalSystems { get; init; }
|
||||
/// <summary>
|
||||
/// The secrets referenced by this site (present, as a content digest, or missing, as
|
||||
/// a marker entry), sorted by name then by missing-last; null when there are none.
|
||||
/// </summary>
|
||||
public List<HashableSecret>? Secrets { get; init; }
|
||||
/// <summary>
|
||||
/// The system-wide shared scripts, sorted by name; null when there are none.
|
||||
/// </summary>
|
||||
public List<HashableSharedScript>? SharedScripts { get; init; }
|
||||
@@ -299,6 +351,34 @@ public sealed class ArtifactRevisionHashService
|
||||
public int TimeoutSeconds { get; init; }
|
||||
}
|
||||
|
||||
private sealed record HashableSecret
|
||||
{
|
||||
/// <summary>
|
||||
/// SHA-256 digest over the secret's ciphertext, nonce, tag, wrapped DEK, wrap-nonce,
|
||||
/// and wrap-tag — never the plaintext. Null for a <see cref="Missing"/> marker entry.
|
||||
/// </summary>
|
||||
public string? ContentDigest { get; init; }
|
||||
/// <summary>
|
||||
/// The key-encryption-key identifier the secret is wrapped under. Null for a
|
||||
/// <see cref="Missing"/> marker entry.
|
||||
/// </summary>
|
||||
public string? KekId { get; init; }
|
||||
/// <summary>
|
||||
/// True when this entry is a <c>MissingSecrets</c> marker (a referenced secret that
|
||||
/// failed to resolve at central) rather than a resolved secret; null when resolved,
|
||||
/// so the same name present vs. missing always serializes — and hashes — differently.
|
||||
/// </summary>
|
||||
public bool? Missing { get; init; }
|
||||
/// <summary>
|
||||
/// The secret's name — the key data connection configuration references it by.
|
||||
/// </summary>
|
||||
public required string Name { get; init; }
|
||||
/// <summary>
|
||||
/// The secret's revision number. Null for a <see cref="Missing"/> marker entry.
|
||||
/// </summary>
|
||||
public long? Revision { get; init; }
|
||||
}
|
||||
|
||||
private sealed record HashableSharedScript
|
||||
{
|
||||
/// <summary>
|
||||
|
||||
@@ -59,6 +59,17 @@ public static class ServiceCollectionExtensions
|
||||
services.AddScoped<DeploymentService>();
|
||||
services.AddScoped<ArtifactDeploymentService>();
|
||||
|
||||
// Resolves a site's referenced secrets (ciphertext rows) against the central
|
||||
// ISecretStore for the artifact deploy (design §3.1/§3.3). Singleton — it holds no
|
||||
// state and the store it wraps is itself registered as a singleton
|
||||
// (SecretsRegistration.AddScadaBridgeSecrets). Depends on ISecretStore, which the
|
||||
// Central composition root always registers (Program.cs calls AddScadaBridgeSecrets
|
||||
// before AddDeploymentManager's registrations are resolved) — a composition that
|
||||
// builds ArtifactDeploymentService without ISecretStore registered is a wiring bug,
|
||||
// not a supported configuration, so this deliberately does NOT fall back to a
|
||||
// null/no-op store.
|
||||
services.AddSingleton<SiteSecretsCollector>();
|
||||
|
||||
// Deterministic hash of a per-site artifact set, stamped on the Site row after a
|
||||
// successful artifact deploy and recomputed to answer "are this site's artifacts
|
||||
// current?". Singleton — it holds no state and touches no DbContext.
|
||||
|
||||
@@ -0,0 +1,68 @@
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Secrets;
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.DeploymentManager;
|
||||
|
||||
/// <summary>
|
||||
/// Result of <see cref="SiteSecretsCollector.CollectAsync"/>: the resolved ciphertext rows for a
|
||||
/// site's referenced secrets, and the references that could not be resolved (absent or
|
||||
/// tombstoned) at central.
|
||||
/// </summary>
|
||||
public sealed record SiteSecretsCollection(
|
||||
IReadOnlyList<SecretArtifact> Secrets,
|
||||
IReadOnlyList<SecretReference> Missing);
|
||||
|
||||
/// <summary>
|
||||
/// Central half of design §3.1/§3.3: derives which secrets a site's data connections reference
|
||||
/// (via <see cref="SecretReferenceScanner"/>) and resolves each one against the central
|
||||
/// <see cref="ISecretStore"/>, verbatim, as its ciphertext <see cref="StoredSecret"/> row. Never
|
||||
/// decrypts and never sees plaintext — the store's <see cref="ISecretStore.GetAsync"/> already
|
||||
/// returns only the encrypted representation.
|
||||
/// </summary>
|
||||
public sealed class SiteSecretsCollector(ISecretStore store)
|
||||
{
|
||||
/// <summary>
|
||||
/// Scans <paramref name="connections"/> for <c>secret:<name></c> references and
|
||||
/// resolves each distinct name against the central secret store.
|
||||
/// </summary>
|
||||
/// <param name="connections">The site's data-connection artifacts to scan.</param>
|
||||
/// <param name="ct">A token to cancel the operation.</param>
|
||||
/// <returns>
|
||||
/// The resolved rows (mapped field-for-field from <see cref="StoredSecret"/>) and the
|
||||
/// references that were absent or tombstoned in the store.
|
||||
/// </returns>
|
||||
public async Task<SiteSecretsCollection> CollectAsync(
|
||||
IEnumerable<DataConnectionArtifact> connections, CancellationToken ct)
|
||||
{
|
||||
var secrets = new List<SecretArtifact>();
|
||||
var missing = new List<SecretReference>();
|
||||
|
||||
foreach (var reference in SecretReferenceScanner.Scan(connections))
|
||||
{
|
||||
var row = await store.GetAsync(new SecretName(reference.Name), ct);
|
||||
if (row is null || row.IsDeleted)
|
||||
{
|
||||
missing.Add(reference);
|
||||
continue;
|
||||
}
|
||||
|
||||
secrets.Add(new SecretArtifact(
|
||||
row.Name.Value,
|
||||
row.ContentType.ToString(),
|
||||
row.Ciphertext,
|
||||
row.Nonce,
|
||||
row.Tag,
|
||||
row.WrappedDek,
|
||||
row.WrapNonce,
|
||||
row.WrapTag,
|
||||
row.KekId,
|
||||
row.Revision,
|
||||
row.CreatedUtc,
|
||||
row.UpdatedUtc,
|
||||
row.Description));
|
||||
}
|
||||
|
||||
return new SiteSecretsCollection(secrets, missing);
|
||||
}
|
||||
}
|
||||
+1
@@ -12,6 +12,7 @@
|
||||
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" />
|
||||
<PackageReference Include="Microsoft.Extensions.Options" />
|
||||
<PackageReference Include="ZB.MOM.WW.Configuration" />
|
||||
<PackageReference Include="ZB.MOM.WW.Secrets.Abstractions" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
|
||||
@@ -836,13 +836,20 @@ akka {{
|
||||
// it after an artifact deploy applies external-system changes.
|
||||
var externalSystemCache = _serviceProvider
|
||||
.GetService<ZB.MOM.WW.ScadaBridge.SiteRuntime.Repositories.ExternalSystemDefinitionCache>();
|
||||
// Task 10 (site-secrets artifact deploy): the same per-node singleton AddSiteRuntime
|
||||
// registers (and AddHostedService runs periodically) is reconciled SYNCHRONOUSLY by
|
||||
// HandleDeployArtifacts right after it writes a shipped secret set, so the deploy
|
||||
// reply's SecretsApplied reflects this node's own apply/verify outcome immediately
|
||||
// rather than waiting for the next periodic pass.
|
||||
var secretsMirror = _serviceProvider
|
||||
.GetService<ZB.MOM.WW.ScadaBridge.SiteRuntime.Secrets.DeployedSecretsMirror>();
|
||||
var dm = SingletonRegistrar.Start(
|
||||
_actorSystem!, "deployment-manager",
|
||||
Props.Create(() => new DeploymentManagerActor(
|
||||
storage, compilationService, sharedScriptLibrary, streamManager,
|
||||
siteRuntimeOptionsValue, dmLogger, dclManager,
|
||||
siteHealthCollector, _serviceProvider, null, deploymentConfigFetcher,
|
||||
null, null, externalSystemCache)),
|
||||
null, null, externalSystemCache, null, null, null, secretsMirror)),
|
||||
_logger, role: siteRole);
|
||||
var dmProxy = dm.Proxy;
|
||||
|
||||
|
||||
@@ -33,9 +33,6 @@ using ZB.MOM.WW.ScadaBridge.Transport;
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
using ZB.MOM.WW.Secrets.Configuration;
|
||||
using ZB.MOM.WW.Secrets.DependencyInjection;
|
||||
using ZB.MOM.WW.Secrets.Replicator.SqlServer;
|
||||
using ZB.MOM.WW.Secrets.Replicator.SqlServer.DependencyInjection;
|
||||
using ZB.MOM.WW.Secrets.Sqlite;
|
||||
using ZB.MOM.WW.Secrets.Ui;
|
||||
using ZB.MOM.WW.Telemetry;
|
||||
using Serilog;
|
||||
@@ -77,30 +74,34 @@ var configuration = new ConfigurationBuilder()
|
||||
// Expand ${secret:...} config references before any validator/binder sees them (Layer A).
|
||||
// Throwaway provider — disposed here, shares no singletons with the host container.
|
||||
//
|
||||
// The expander must read the SAME store the running node will serve. On a central node in Grpc
|
||||
// replication mode that store is the SHARED SQL-Server store (see SecretsRegistration,
|
||||
// scadaproj#4) — an expander left on SQLite there would resolve pre-host ${secret:} references
|
||||
// from a stale/empty local store, silently diverging from what the node's own hub serves. Every
|
||||
// other case (sites, SqlServer mode, replication off) keeps the local SQLite path exactly as it
|
||||
// always was. A blank or ${secret:}-valued connection string fails HERE, through the same
|
||||
// EnsureCentralSharedStoreConnectionString check AddScadaBridgeSecrets runs — this is the
|
||||
// earliest code that would otherwise hand the bad value to SqlConnection, whose generic
|
||||
// "initialization string" format error would bury the designed message.
|
||||
// The expander MUST read the SAME store the running node will serve — SecretsRegistration.
|
||||
// UsesCentralSharedStore is the single predicate deciding that, shared with the runtime
|
||||
// registration in AddScadaBridgeSecrets, so the two can never disagree. On a Central node with
|
||||
// Secrets:Store:Provider=SqlServer that is the SHARED SQL-Server store (Task 5): an expander left
|
||||
// on SQLite there would resolve pre-host ${secret:} references (e.g. ScadaBridge:Security:
|
||||
// JwtSigningKey) from a store the node does not actually serve, so a value rotated through the
|
||||
// shared store would never reach the pre-host config, and two central nodes could disagree on
|
||||
// what a token resolves to. Every other case (sites, the default per-node SQLite store) keeps the
|
||||
// local SQLite path exactly as it always was. A blank or ${secret:}-valued connection string on
|
||||
// the shared-store path fails HERE, through the same EnsureCentralSharedStoreConnectionString
|
||||
// check AddScadaBridgeSecrets runs — this is the earliest code that would otherwise hand the bad
|
||||
// value to SqlConnection, whose generic "initialization string" format error would bury the
|
||||
// designed message.
|
||||
//
|
||||
// Two consequences worth knowing. This block runs BEFORE Serilog exists and outside the
|
||||
// try/catch below, so a SQL Server outage at central boot exits with a bare stderr stack
|
||||
// trace, no structured log — honest and container-restart-retryable, just unenriched; the
|
||||
// SQLite path had the same window but its only failure mode was a local file. (That exit
|
||||
// is real only because of the UnhandledException handler at the top of this file — the
|
||||
// runtime's own crash path wedges as container PID 1, ScadaBridge#34.) And the
|
||||
// role check is deliberately NOT folded into SecretsRegistration: that class refuses to
|
||||
// read the node role from configuration (a config-read role could silently turn a site
|
||||
// into a hub — see SecretsNodeRole), while Program.cs must read it to branch the whole
|
||||
// composition anyway, so the config read stays here at the composition root.
|
||||
var expanderUsesSharedSqlStore =
|
||||
string.Equals(
|
||||
configuration["ScadaBridge:Node:Role"], "Central", StringComparison.OrdinalIgnoreCase)
|
||||
&& SecretsRegistration.UsesGrpcHub(configuration);
|
||||
// runtime's own crash path wedges as container PID 1, ScadaBridge#34.) And the role read
|
||||
// stays here at the composition root rather than inside SecretsRegistration: that class
|
||||
// refuses to read the node role from configuration (a config-read role could silently turn a
|
||||
// site into a shared-store central — see SecretsNodeRole), while Program.cs must read it to
|
||||
// branch the whole composition anyway.
|
||||
var expanderRole = string.Equals(
|
||||
configuration["ScadaBridge:Node:Role"], "Central", StringComparison.OrdinalIgnoreCase)
|
||||
? SecretsNodeRole.Central
|
||||
: SecretsNodeRole.Site;
|
||||
|
||||
// Secrets 0.6.2 validates Secrets:SqlitePath as absolute AND outside the content root, finding the
|
||||
// content root from IHostEnvironment — which this throwaway ServiceCollection does not have, and
|
||||
@@ -109,31 +110,18 @@ var expanderUsesSharedSqlStore =
|
||||
// AddPreHostSqliteExpander for why the content-root argument is load-bearing and silent when
|
||||
// dropped, and PreHostSecretsContentRootTests for the regression guard.
|
||||
var expanderServices = new ServiceCollection();
|
||||
if (expanderUsesSharedSqlStore)
|
||||
{
|
||||
SecretsRegistration.EnsureCentralSharedStoreConnectionString(configuration);
|
||||
expanderServices.AddZbSecretsSqlServerStore(configuration, "Secrets");
|
||||
}
|
||||
else
|
||||
{
|
||||
expanderServices.AddPreHostSqliteExpander(
|
||||
configuration, SecretsRegistration.ResolveHostContentRoot());
|
||||
}
|
||||
expanderServices.AddPreHostExpander(
|
||||
configuration, expanderRole, SecretsRegistration.ResolveHostContentRoot());
|
||||
|
||||
#pragma warning disable ASP0000 // deliberate throwaway container
|
||||
await using (var secretsProvider = expanderServices.BuildServiceProvider())
|
||||
#pragma warning restore ASP0000
|
||||
{
|
||||
if (expanderUsesSharedSqlStore)
|
||||
{
|
||||
await secretsProvider.GetRequiredService<SqlServerSecretsStoreMigrator>()
|
||||
.MigrateAsync(default);
|
||||
}
|
||||
else
|
||||
{
|
||||
await secretsProvider.GetRequiredService<SqliteSecretsStoreMigrator>()
|
||||
.MigrateAsync(default);
|
||||
}
|
||||
// ISecretsStoreMigrator is the store-agnostic seam both AddZbSecrets (SQLite) and
|
||||
// AddZbSecretsSqlServerStore register — resolving through it means this block does not need
|
||||
// to know, or re-derive, which concrete store AddPreHostExpander just composed.
|
||||
await secretsProvider.GetRequiredService<ISecretsStoreMigrator>()
|
||||
.MigrateAsync(default);
|
||||
|
||||
var resolver = secretsProvider.GetRequiredService<ISecretResolver>();
|
||||
await new SecretReferenceExpander(resolver)
|
||||
@@ -397,14 +385,14 @@ try
|
||||
.GetValue<bool>(nameof(ZB.MOM.WW.ScadaBridge.Security.Auth.AuthDisableLoginOptions.AllowOutsideDevelopment)));
|
||||
builder.Services.AddSecurity(disableLogin);
|
||||
builder.Services.AddCentralUI();
|
||||
// Local SQLite store by default. With Secrets:Replication:Enabled true it becomes either a
|
||||
// local store synced against a shared SQL-Server hub (Mode=SqlServer, and only when a hub
|
||||
// connection string is present) or — the production topology, scadaproj#3 — the HOST half
|
||||
// of the pull-only gRPC secrets hub (Mode=Grpc), which central serves and sites sweep.
|
||||
// The Central role never registers the sweep client; passing the role explicitly is what
|
||||
// makes that a compile-time property of this composition root rather than a config guess.
|
||||
// See SecretsRegistration for why the SQL gate exists (eager options validation), why the
|
||||
// hub credential can never itself come from the hub, and why gRPC mode fails closed.
|
||||
// Local SQLite store by default. With Secrets:Store:Provider=SqlServer this central node's
|
||||
// store becomes the SHARED SQL-Server database instead — no hub, no follower sweep, just
|
||||
// both central nodes reading and writing the same rows, so a clustered pair can never ship
|
||||
// sites a drifted secret set (Task 5). The Layer-A pre-host expander above resolves this
|
||||
// same predicate (SecretsRegistration.UsesCentralSharedStore) so it always reads the store
|
||||
// this node actually serves. See SecretsRegistration for why the SQL gate exists (eager
|
||||
// options validation) and why the connection string can never itself be a ${secret:}
|
||||
// reference.
|
||||
builder.Services.AddScadaBridgeSecrets(builder.Configuration, SecretsNodeRole.Central);
|
||||
// Secrets UI authorization: adds the named policies secrets:manage + secrets:reveal
|
||||
// (role-based) consumed by the /admin/secrets page. AddSecretsAuthorization only ADDS
|
||||
@@ -739,21 +727,9 @@ try
|
||||
// connection to the right pipeline by listener/protocol.
|
||||
app.MapGrpcService<ZB.MOM.WW.ScadaBridge.Communication.Grpc.CentralControlGrpcService>();
|
||||
|
||||
// The pull-only secrets hub (scadaproj#3), on the SAME h2c listener and the same addressing
|
||||
// convention — a site dials central's CentralGrpcPort, never central's database. It shares
|
||||
// the listener with CentralControlService the way the site's LocalDb sync endpoint shares
|
||||
// its one: two disjoint service prefixes, two independent fail-closed gates. The
|
||||
// CentralControlAuthInterceptor registered on AddGrpc is prefix-scoped to
|
||||
// CentralControlService and passes hub calls straight through; the hub's own
|
||||
// SecretsHubAuthInterceptor — attached per-service by AddZbSecretsGrpcHub — is what gates
|
||||
// it, on the shared Secrets:GrpcHub:BearerToken.
|
||||
//
|
||||
// A NO-OP unless this node registered the hub. That is not an optimisation: mapping the
|
||||
// service without registering it would map a hub whose interceptor was never attached,
|
||||
// i.e. an unauthenticated endpoint serving every secret central holds. Both this call and
|
||||
// the registration above route through SecretsRegistration.UsesGrpcHub so they cannot
|
||||
// drift apart.
|
||||
app.MapScadaBridgeSecretsHub(app.Configuration);
|
||||
// The pull-only secrets hub that used to be mapped here (scadaproj#3) is retired — Task 11.
|
||||
// Sites receive secrets exclusively via Deploy Artifacts; there is no hub endpoint to map
|
||||
// on central and nothing left for a site to sweep. See SecretsRegistration.
|
||||
|
||||
app.MapStaticAssets();
|
||||
app.MapCentralUI<ZB.MOM.WW.ScadaBridge.Host.Components.App>();
|
||||
|
||||
@@ -1,45 +1,26 @@
|
||||
using Microsoft.AspNetCore.Routing;
|
||||
using Microsoft.Extensions.Hosting.WindowsServices;
|
||||
using ZB.MOM.WW.Secrets.DependencyInjection;
|
||||
using ZB.MOM.WW.Secrets.Replicator.Grpc.DependencyInjection;
|
||||
using ZB.MOM.WW.Secrets.Replicator.SqlServer.DependencyInjection;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Host;
|
||||
|
||||
/// <summary>
|
||||
/// Which half of a clustered secrets topology this node composes. Passed in by the caller rather
|
||||
/// than read from configuration, because both composition roots already know statically which one
|
||||
/// they are — <c>Program.cs</c> for central and <see cref="SiteServiceRegistration"/> for a site —
|
||||
/// and a role read from a key could be got wrong in a way that silently turns a site into a hub.
|
||||
/// Which half of a composition this node is. Passed in by the caller rather than read from
|
||||
/// configuration, because both composition roots already know statically which one they are —
|
||||
/// <c>Program.cs</c> for central and <see cref="SiteServiceRegistration"/> for a site — and a role
|
||||
/// read from a key could be got wrong in a way that silently turns a site's secret handling into
|
||||
/// central's.
|
||||
/// </summary>
|
||||
public enum SecretsNodeRole
|
||||
{
|
||||
/// <summary>The central cluster. Hosts the gRPC secrets hub; never sweeps from one.</summary>
|
||||
/// <summary>The central cluster.</summary>
|
||||
Central,
|
||||
|
||||
/// <summary>A site cluster. Sweeps from central's hub; never hosts one.</summary>
|
||||
/// <summary>A site cluster.</summary>
|
||||
Site,
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The clustered-replication transport, selected by <c>Secrets:Replication:Mode</c>.
|
||||
/// </summary>
|
||||
public enum SecretsReplicationMode
|
||||
{
|
||||
/// <summary>
|
||||
/// Local store bidirectionally synced with a shared SQL-Server hub. The historical mode and
|
||||
/// therefore the default, so an existing configuration that sets only
|
||||
/// <c>Secrets:Replication:Enabled</c> keeps behaving exactly as it did.
|
||||
/// </summary>
|
||||
SqlServer,
|
||||
|
||||
/// <summary>
|
||||
/// Local store on every node, converged by a pull-only gRPC hub that central hosts. The
|
||||
/// production topology (scadaproj#3).
|
||||
/// </summary>
|
||||
Grpc,
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Single composition-root entry point for the host container's secret store, shared by the
|
||||
/// central-role registrations in <c>Program.cs</c> and the site-role registrations in
|
||||
@@ -47,61 +28,44 @@ public enum SecretsReplicationMode
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// Clustered replication is <b>opt-in and off by default</b>. ScadaBridge is hub-and-spoke — one
|
||||
/// central cluster plus N separate site clusters — and every supported topology gives each SITE
|
||||
/// node a LOCAL store, so a site keeps resolving secrets straight through a WAN outage. What
|
||||
/// differs between the modes is how convergence works, and — in gRPC mode — where central's own
|
||||
/// store lives.
|
||||
/// <b>Task 11 retired ALL secrets replication in ScadaBridge</b> — both the pull-only gRPC hub
|
||||
/// (central hosts, sites sweep) and the shared-SQL-Server <c>ReplicatingSecretStore</c> path that
|
||||
/// preceded it. ScadaBridge is hub-and-spoke — one central cluster plus N separate site clusters —
|
||||
/// and every topology now gives each SITE node a permanently LOCAL, undecorated SQLite store, so a
|
||||
/// site keeps resolving secrets straight through a WAN outage AND never republishes anything it
|
||||
/// deletes. That second property is load-bearing, not incidental:
|
||||
/// <c>ZB.MOM.WW.ScadaBridge.SiteRuntime.Secrets.DeployedSecretsMirror</c> deletes a name from the
|
||||
/// local store the moment a deploy artifact stops referencing it, and a replication-decorated
|
||||
/// store (the retired SQL-Server mode) would have published that delete to every other node
|
||||
/// sharing the hub — turning a single site's unreferencing of a secret into a fleet-wide deletion.
|
||||
/// Sites now receive secrets exclusively via Deploy Artifacts (Tasks 1–10); there is no sweep, no
|
||||
/// hub, and nothing for a site to converge against but its own local file.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b><see cref="SecretsReplicationMode.Grpc"/> is the production topology</b> (scadaproj#3):
|
||||
/// central hosts a pull-only gRPC hub and each site node sweeps it. It exists because
|
||||
/// <see cref="SecretsReplicationMode.SqlServer"/> requires every site node to hold a connection
|
||||
/// string to central's database, which violates ScadaBridge's standing rule that sites talk to
|
||||
/// central, not to central's DB. Pull-only is a property of the wire contract rather than a
|
||||
/// setting: the proto carries no write RPC, so a site cannot push even if misconfigured, and
|
||||
/// secrets therefore originate at central only. <b>Central's own store in this mode is the SHARED
|
||||
/// SQL-Server store</b> (<c>AddZbSecretsSqlServerStore</c>), not local SQLite: both central nodes
|
||||
/// read and write ONE copy of every row, so the two hub instances serve identical manifests by
|
||||
/// construction. The alternative — an independent local store per central node — is the recorded
|
||||
/// defect (scadaproj#4): one central answering authenticated followers with an empty manifest,
|
||||
/// turning site-side hub failover into a silent convergence stop. The accepted trade is that
|
||||
/// central secret resolution now depends on SQL Server availability — acceptable because central
|
||||
/// is already SQL-coupled to its core, and a SQL outage makes the hub <em>throw</em> (sites keep
|
||||
/// last-known-good), never serve empty. The pre-host <c>${secret:}</c> expander in
|
||||
/// <c>Program.cs</c> follows this store swap, so pre-host references resolve from the same store
|
||||
/// the running node serves.
|
||||
/// <b><see cref="StoreProviderKey"/> is a SEPARATE, still-supported feature and is unaffected by
|
||||
/// the retirement above.</b> A fleet that ships secrets to sites on every artifact deploy still
|
||||
/// needs a clustered CENTRAL pair to see the same secret set, so <c>Secrets:Store:Provider=SqlServer</c>
|
||||
/// points both central nodes at ONE shared SQL-Server database directly — no hub, no follower
|
||||
/// sweep, just two nodes reading and writing the same rows. This is CENTRAL-only; a Site node
|
||||
/// always ignores the key and keeps its local SQLite store.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>The SQL-Server gate is not a style preference.</b> <c>AddZbSecretsSqlServerReplication</c>
|
||||
/// <b>The SQL-Server gate is not a style preference.</b> <c>AddZbSecretsSqlServerStore</c>
|
||||
/// validates its options EAGERLY at registration time, so calling it unconditionally would throw at
|
||||
/// startup on every node where <c>Secrets:SqlServer:ConnectionString</c> is unset — which is every
|
||||
/// dev box, every docker node and every existing deployment. Both the explicit
|
||||
/// <c>Secrets:Replication:Enabled</c> flag and a non-blank connection string are therefore required
|
||||
/// dev box, every docker node and every existing deployment. Both
|
||||
/// <c>Secrets:Store:Provider=SqlServer</c> and a non-blank connection string are therefore required
|
||||
/// before the SQL-Server package is touched at all; otherwise this is byte-for-byte the plain
|
||||
/// local-SQLite registration it has always been.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>The gRPC mode deliberately does NOT degrade the same way.</b> A missing connection string in
|
||||
/// SQL-Server mode falls back to a local-only store with a loud warning, and that behaviour is
|
||||
/// preserved untouched. gRPC mode instead fails at startup when its endpoint or bearer token is
|
||||
/// missing — or, on central, when the shared-store connection string is — because the fallback is
|
||||
/// the exact outcome the hub exists to prevent — a site serving secrets that silently never
|
||||
/// converge with central — and because selecting the mode takes an explicit second key, so an
|
||||
/// operator who typed it meant it.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Bootstrap constraint.</b> A replication credential can never itself come from the replicated
|
||||
/// set — a node cannot read the hub to learn how to reach the hub. In SQL-Server mode the hub
|
||||
/// connection string must arrive from outside it: an environment variable, or a <c>${secret:}</c>
|
||||
/// reference seeded in that node's own LOCAL store (the pre-host expander in <c>Program.cs</c>
|
||||
/// runs against the local SQLite store before the host container exists, so such a reference does
|
||||
/// resolve). In gRPC mode the rule is stricter on central: <c>Secrets:SqlServer:ConnectionString</c>
|
||||
/// can never be a <c>${secret:}</c> reference at all, because on central the pre-host expander
|
||||
/// itself runs against the shared SQL store (see <c>Program.cs</c>) and would need this very
|
||||
/// string to reach the store that resolves it — registration rejects such a value outright. The
|
||||
/// same holds for <c>Secrets:GrpcHub:BearerToken</c>, which is supplied from appsettings or the
|
||||
/// environment exactly as the mesh pre-shared keys are.
|
||||
/// <b>Bootstrap constraint.</b> The shared store's own connection string can never itself come from
|
||||
/// that store — a node cannot read the database to learn how to reach the database. It must arrive
|
||||
/// from outside it: an environment variable, or a literal value in appsettings. It can never be a
|
||||
/// <c>${secret:}</c> reference, because on central the pre-host expander (<see cref="AddPreHostExpander"/>,
|
||||
/// called from <c>Program.cs</c> — see <see cref="UsesCentralSharedStore"/>) resolves EVERY
|
||||
/// <c>${secret:}</c> token in configuration, including this one, from the very store this string
|
||||
/// points at — the reference could never resolve, so registration rejects it outright.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public static class SecretsRegistration
|
||||
@@ -109,81 +73,43 @@ public static class SecretsRegistration
|
||||
/// <summary>Configuration section holding the core secrets options.</summary>
|
||||
public const string SecretsSectionPath = "Secrets";
|
||||
|
||||
/// <summary>Configuration section holding the SQL-Server hub options.</summary>
|
||||
/// <summary>Configuration section holding the shared SQL-Server store options.</summary>
|
||||
public const string SqlServerSectionPath = "Secrets:SqlServer";
|
||||
|
||||
/// <summary>
|
||||
/// Configuration section holding the gRPC hub options. One section serves both halves: central
|
||||
/// reads <c>BearerToken</c> + <c>MaxNamesPerRequest</c> from it, a site reads <c>Endpoint</c>,
|
||||
/// <c>BearerToken</c> and the sweep timings.
|
||||
/// </summary>
|
||||
public const string GrpcHubSectionPath = "Secrets:GrpcHub";
|
||||
|
||||
/// <summary>Configuration key gating clustered replication. Absent or false = off.</summary>
|
||||
public const string ReplicationEnabledKey = "Secrets:Replication:Enabled";
|
||||
|
||||
/// <summary>
|
||||
/// Configuration key selecting the replication transport. Absent or blank =
|
||||
/// <see cref="SecretsReplicationMode.SqlServer"/>, the historical behaviour.
|
||||
/// </summary>
|
||||
public const string ReplicationModeKey = "Secrets:Replication:Mode";
|
||||
|
||||
/// <summary>Configuration key holding the shared SQL-Server hub connection string.</summary>
|
||||
/// <summary>Configuration key holding the shared SQL-Server store's connection string.</summary>
|
||||
public const string HubConnectionStringKey = "Secrets:SqlServer:ConnectionString";
|
||||
|
||||
/// <summary>
|
||||
/// Reads <see cref="ReplicationModeKey"/>. An absent or blank value is
|
||||
/// <see cref="SecretsReplicationMode.SqlServer"/>; anything unrecognised throws.
|
||||
/// Configuration key selecting a CENTRAL node's own local store provider. Absent or blank =
|
||||
/// <c>Sqlite</c> (per-node, the historical default); <c>SqlServer</c> points both central
|
||||
/// nodes at ONE shared SQL-Server database directly, with no hub in between — for a fleet
|
||||
/// that ships secrets to sites on every artifact deploy, a clustered central needs both nodes
|
||||
/// to see the same secret set. A Site node always ignores this key.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The value is validated whenever it is present, even with replication switched off. A typo
|
||||
/// only surfaces at the boot that introduces it that way — deferring the check to whenever
|
||||
/// somebody later flips <see cref="ReplicationEnabledKey"/> would fail a node in an unrelated
|
||||
/// change, pointing at a key nobody touched.
|
||||
/// </remarks>
|
||||
/// <param name="config">Application configuration.</param>
|
||||
/// <returns>The selected mode.</returns>
|
||||
/// <exception cref="InvalidOperationException">The configured value is not a known mode.</exception>
|
||||
public static SecretsReplicationMode ResolveReplicationMode(IConfiguration config)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(config);
|
||||
|
||||
var raw = config[ReplicationModeKey];
|
||||
if (string.IsNullOrWhiteSpace(raw))
|
||||
{
|
||||
return SecretsReplicationMode.SqlServer;
|
||||
}
|
||||
|
||||
if (!Enum.TryParse(raw.Trim(), ignoreCase: true, out SecretsReplicationMode mode)
|
||||
|| !Enum.IsDefined(mode))
|
||||
{
|
||||
throw new InvalidOperationException(
|
||||
$"{ReplicationModeKey} is '{raw}', which is not a known secrets replication mode. "
|
||||
+ $"Valid values are '{nameof(SecretsReplicationMode.SqlServer)}' and "
|
||||
+ $"'{nameof(SecretsReplicationMode.Grpc)}'.");
|
||||
}
|
||||
|
||||
return mode;
|
||||
}
|
||||
public const string StoreProviderKey = "Secrets:Store:Provider";
|
||||
|
||||
/// <summary>
|
||||
/// Whether this node participates in the pull-only gRPC secrets hub.
|
||||
/// Whether this node's secret store is the shared SQL-Server database — the single predicate
|
||||
/// behind BOTH <see cref="AddScadaBridgeSecrets"/>'s runtime registration AND
|
||||
/// <see cref="AddPreHostExpander"/>'s Layer-A <c>${secret:}</c> expander, so the two can never
|
||||
/// disagree about which store a Central node serves. That matters more than tidiness: an
|
||||
/// expander that resolved boot-time <c>${secret:}</c> tokens from the per-node SQLite file
|
||||
/// while the running node served the shared SQL store would let two central nodes boot with
|
||||
/// different resolved values (e.g. a different <c>JwtSigningKey</c>) — exactly the per-node
|
||||
/// drift <see cref="StoreProviderKey"/>=SqlServer exists to prevent.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The single predicate behind BOTH the service registration and
|
||||
/// <see cref="MapScadaBridgeSecretsHub"/>, so the two can never disagree. That matters more
|
||||
/// than tidiness: mapping the hub endpoint without registering it would map a service whose
|
||||
/// fail-closed bearer interceptor was never attached — an unauthenticated hub serving every
|
||||
/// secret central holds.
|
||||
/// </remarks>
|
||||
/// <param name="config">Application configuration.</param>
|
||||
/// <returns><see langword="true"/> when replication is enabled in gRPC mode.</returns>
|
||||
public static bool UsesGrpcHub(IConfiguration config)
|
||||
/// <param name="role">Which half of the composition this node is.</param>
|
||||
/// <returns>
|
||||
/// <see langword="true"/> when <paramref name="role"/> is <see cref="SecretsNodeRole.Central"/>
|
||||
/// and <see cref="StoreProviderKey"/> is (case-insensitively) <c>SqlServer</c>.
|
||||
/// </returns>
|
||||
internal static bool UsesCentralSharedStore(IConfiguration config, SecretsNodeRole role)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(config);
|
||||
|
||||
return config.GetValue<bool>(ReplicationEnabledKey)
|
||||
&& ResolveReplicationMode(config) == SecretsReplicationMode.Grpc;
|
||||
return role == SecretsNodeRole.Central
|
||||
&& string.Equals(config[StoreProviderKey], "SqlServer", StringComparison.OrdinalIgnoreCase);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -246,54 +172,112 @@ public static class SecretsRegistration
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The one definition of "a Central node in Grpc mode has a usable shared-store connection
|
||||
/// string", shared by <see cref="AddScadaBridgeSecrets"/> and the Layer-A expander in
|
||||
/// <c>Program.cs</c> so the two can never diverge. The expander runs FIRST on a real boot —
|
||||
/// without this sharing, a bad value would die inside <c>SqlConnection</c> with a generic
|
||||
/// "initialization string" format error before the designed message ever fired.
|
||||
/// Composes the Layer-A pre-host <c>${secret:}</c> expander stack for <paramref name="role"/> —
|
||||
/// the shared SQL-Server store when <see cref="UsesCentralSharedStore"/> is true, otherwise the
|
||||
/// plain local SQLite store (<see cref="AddPreHostSqliteExpander"/>).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// This is the fix for the gap <see cref="UsesCentralSharedStore"/>'s own remarks describe: the
|
||||
/// expander MUST read the same store <see cref="AddScadaBridgeSecrets"/> will register for the
|
||||
/// running node, or a boot-time <c>${secret:}</c> reference (e.g. in
|
||||
/// <c>ScadaBridge:Security:JwtSigningKey</c>) resolves from a store the node does not actually
|
||||
/// serve. Both this method and <see cref="AddScadaBridgeSecrets"/> branch on the exact same
|
||||
/// <see cref="UsesCentralSharedStore"/> call, so they cannot drift apart the way the old
|
||||
/// Grpc-mode expander branch in <c>Program.cs</c> and the Grpc-mode runtime branch here once
|
||||
/// had to be kept in step by hand.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// On the shared-store path this also runs <see cref="EnsureCentralSharedStoreConnectionString"/>
|
||||
/// BEFORE composing anything — the same fail-closed rules apply here as at runtime: a blank
|
||||
/// connection string, or one holding a <c>${secret:}</c> reference (which could never resolve;
|
||||
/// this very expander is what would have to resolve it), fails the boot with the designed
|
||||
/// message instead of a generic SqlConnection format error.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
/// <param name="services">The throwaway expander service collection.</param>
|
||||
/// <param name="config">Application configuration.</param>
|
||||
/// <param name="role">Which half of the composition this node is.</param>
|
||||
/// <param name="contentRoot">The host's content root, from <see cref="ResolveHostContentRoot"/> — only used on the SQLite path.</param>
|
||||
/// <returns>The same collection, for chaining.</returns>
|
||||
/// <exception cref="InvalidOperationException">
|
||||
/// <paramref name="role"/> is Central with the shared store selected, and
|
||||
/// <see cref="HubConnectionStringKey"/> is blank or a <c>${secret:}</c> reference.
|
||||
/// </exception>
|
||||
public static IServiceCollection AddPreHostExpander(
|
||||
this IServiceCollection services,
|
||||
IConfiguration config,
|
||||
SecretsNodeRole role,
|
||||
string contentRoot)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(services);
|
||||
ArgumentNullException.ThrowIfNull(config);
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(contentRoot);
|
||||
|
||||
if (UsesCentralSharedStore(config, role))
|
||||
{
|
||||
EnsureCentralSharedStoreConnectionString(config, $"{StoreProviderKey}=SqlServer");
|
||||
|
||||
// Same 3-arg overload as the runtime registration in AddScadaBridgeSecrets — this is
|
||||
// deliberately the SAME store, not a lookalike one composed independently.
|
||||
return services.AddZbSecretsSqlServerStore(
|
||||
config, SecretsSectionPath, SqlServerSectionPath);
|
||||
}
|
||||
|
||||
return services.AddPreHostSqliteExpander(config, contentRoot);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The one definition of "a Central node has a usable shared-store connection string", shared
|
||||
/// by every caller that puts a Central node on the shared SQL-Server store — the
|
||||
/// <see cref="StoreProviderKey"/>=SqlServer branch of <see cref="AddScadaBridgeSecrets"/> — so
|
||||
/// nothing can diverge on what "usable" means.
|
||||
/// </summary>
|
||||
/// <param name="config">Application configuration.</param>
|
||||
/// <param name="triggerContext">
|
||||
/// Names the setting that put this Central node on the shared store, e.g.
|
||||
/// <c>"Secrets:Store:Provider=SqlServer"</c> — folded into the blank-connection-string
|
||||
/// message so a boot failure names its own real trigger.
|
||||
/// </param>
|
||||
/// <returns>The validated connection string.</returns>
|
||||
/// <exception cref="InvalidOperationException">
|
||||
/// <see cref="HubConnectionStringKey"/> is blank or contains a <c>${secret:}</c> reference.
|
||||
/// </exception>
|
||||
internal static string EnsureCentralSharedStoreConnectionString(IConfiguration config)
|
||||
internal static string EnsureCentralSharedStoreConnectionString(
|
||||
IConfiguration config, string triggerContext)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(config);
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(triggerContext);
|
||||
|
||||
var hubConnectionString = config[HubConnectionStringKey];
|
||||
|
||||
if (string.IsNullOrWhiteSpace(hubConnectionString))
|
||||
{
|
||||
// Fail closed, matching the mode's philosophy: central in Grpc mode REQUIRES the
|
||||
// shared SQL store. An independent local store per central node is exactly the
|
||||
// divergence scadaproj#4 recorded — one central hub answered authenticated followers
|
||||
// with an EMPTY manifest, so a hub failover would "succeed" against nothing and stop
|
||||
// convergence silently.
|
||||
// Fail closed: this configuration REQUIRES the shared SQL store. An independent
|
||||
// local store per central node would silently drift the moment anyone edits a
|
||||
// secret on only one node, then ship an inconsistent secret set to sites on
|
||||
// artifact deploy.
|
||||
throw new InvalidOperationException(
|
||||
$"{HubConnectionStringKey} is empty, but this node is Central with "
|
||||
+ $"{ReplicationModeKey}=Grpc. Central in Grpc mode requires the "
|
||||
+ "SHARED SQL-Server secret store — an independent local store per "
|
||||
+ "central node is the divergence scadaproj#4 recorded (a hub "
|
||||
+ "failover would 'succeed' against an empty manifest). Supply the "
|
||||
+ "connection string via appsettings or the environment "
|
||||
+ "(Secrets__SqlServer__ConnectionString).");
|
||||
+ $"{triggerContext}. That configuration requires the SHARED "
|
||||
+ "SQL-Server secret store — an independent local store per "
|
||||
+ "central node would silently drift between the two nodes. "
|
||||
+ "Supply the connection string via appsettings or the "
|
||||
+ "environment (Secrets__SqlServer__ConnectionString).");
|
||||
}
|
||||
|
||||
if (hubConnectionString.Contains("${secret:", StringComparison.Ordinal))
|
||||
{
|
||||
// The secrets store's own connection string can never be a secret reference: the
|
||||
// pre-host expander needs the store this string points at in order to resolve it —
|
||||
// bootstrap circularity, the same rule the class remarks document for the hub bearer
|
||||
// token. The message deliberately does not echo the configured value: a mistyped
|
||||
// value here could be a pasted real credential.
|
||||
// bootstrap circularity. The message deliberately does not echo the configured
|
||||
// value: a mistyped value here could be a pasted real credential.
|
||||
throw new InvalidOperationException(
|
||||
$"{HubConnectionStringKey} contains a ${{secret:}} reference. The "
|
||||
+ "secret store's own connection string can never be a secret "
|
||||
+ "reference — the pre-host expander needs the store this string "
|
||||
+ "points at in order to resolve it (bootstrap circularity, the "
|
||||
+ "same rule as Secrets:GrpcHub:BearerToken). Supply a literal "
|
||||
+ "value via appsettings or the environment "
|
||||
+ "points at in order to resolve it (bootstrap circularity). "
|
||||
+ "Supply a literal value via appsettings or the environment "
|
||||
+ "(Secrets__SqlServer__ConnectionString).");
|
||||
}
|
||||
|
||||
@@ -301,15 +285,21 @@ public static class SecretsRegistration
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Registers the host container's secret store: a plain local SQLite store by default, or a
|
||||
/// replicating topology when <see cref="ReplicationEnabledKey"/> is set — a shared SQL-Server
|
||||
/// hub, or the pull-only gRPC hub whose half is chosen by <paramref name="role"/>. In gRPC
|
||||
/// mode a central node's store is the SHARED SQL-Server store, and registration fails closed
|
||||
/// when <see cref="HubConnectionStringKey"/> is blank or is a <c>${secret:}</c> reference.
|
||||
/// Registers the host container's secret store: a plain local SQLite store, or — on Central
|
||||
/// only, with <see cref="StoreProviderKey"/> set to <c>SqlServer</c> — the shared SQL-Server
|
||||
/// store both central nodes point at directly. Registration fails closed when
|
||||
/// <see cref="HubConnectionStringKey"/> is blank or is a <c>${secret:}</c> reference.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// A Site node (<paramref name="role"/> = <see cref="SecretsNodeRole.Site"/>) always resolves
|
||||
/// to the plain local SQLite store — there is no replicating or decorated store left for a
|
||||
/// site to be given. This is deliberate (Task 11): <c>DeployedSecretsMirror</c> deletes a name
|
||||
/// from the local store directly, and a replication decorator would have published that
|
||||
/// delete fleet-wide.
|
||||
/// </remarks>
|
||||
/// <param name="services">The service collection to register into.</param>
|
||||
/// <param name="config">Application configuration for options binding.</param>
|
||||
/// <param name="role">Which half of a clustered topology this node composes.</param>
|
||||
/// <param name="role">Which half of the composition this node is.</param>
|
||||
/// <returns>The same <paramref name="services"/> instance, for chaining.</returns>
|
||||
public static IServiceCollection AddScadaBridgeSecrets(
|
||||
this IServiceCollection services,
|
||||
@@ -319,112 +309,24 @@ public static class SecretsRegistration
|
||||
ArgumentNullException.ThrowIfNull(services);
|
||||
ArgumentNullException.ThrowIfNull(config);
|
||||
|
||||
// UsesGrpcHub is the single predicate shared with MapScadaBridgeSecretsHub — the branch
|
||||
// condition must never be re-derived here, or registration and endpoint mapping could
|
||||
// disagree about whether the hub is on.
|
||||
if (UsesGrpcHub(config))
|
||||
// Secrets:Store:Provider=SqlServer on Central only. A Site never reaches this: it always
|
||||
// keeps its local SQLite store, straight through a WAN outage, same as every other
|
||||
// topology, and never decorated — see the class remarks. UsesCentralSharedStore is the
|
||||
// SAME predicate AddPreHostExpander branches on, so the two can never disagree about
|
||||
// which store a Central node serves.
|
||||
if (UsesCentralSharedStore(config, role))
|
||||
{
|
||||
switch (role)
|
||||
{
|
||||
case SecretsNodeRole.Central:
|
||||
// Runs BEFORE any registration so the boot failure names the real problem
|
||||
// instead of surfacing later as a package validation message or a
|
||||
// malformed-connection-string fault on first use. On a real boot the Layer-A
|
||||
// expander in Program.cs has already run the same check even earlier.
|
||||
EnsureCentralSharedStoreConnectionString(config);
|
||||
// Fail-closed rules: blank, or a ${secret:} reference (bootstrap circularity — the
|
||||
// pre-host expander needs this same store to resolve it).
|
||||
EnsureCentralSharedStoreConnectionString(config, $"{StoreProviderKey}=SqlServer");
|
||||
|
||||
// Central's store is the SHARED SQL-Server store: registers
|
||||
// SqlServerSecretStore as ISecretStore BEFORE calling AddZbSecrets internally,
|
||||
// displacing SQLite — do NOT also call AddZbSecrets here. One copy of every
|
||||
// row means both central hubs serve identical manifests by construction, which
|
||||
// is what makes site-side hub failover between them safe.
|
||||
services.AddZbSecretsSqlServerStore(
|
||||
config, SecretsSectionPath, SqlServerSectionPath);
|
||||
|
||||
// Throws when Secrets:GrpcHub:BearerToken is unset. Deliberately not caught: a
|
||||
// hub that starts and refuses every follower looks like a network fault from
|
||||
// the site end, and would be discovered as a stale secret rather than as a
|
||||
// boot failure.
|
||||
services.AddZbSecretsGrpcHub(config, GrpcHubSectionPath);
|
||||
break;
|
||||
case SecretsNodeRole.Site:
|
||||
// A site keeps its LOCAL SQLite store — it resolves secrets straight through a
|
||||
// WAN outage, and sites talk to central, never to central's database. Neither
|
||||
// gRPC extension registers the local store — pull-only means local writes
|
||||
// never leave the node, so there is nothing to decorate and the store stays
|
||||
// exactly as configured. Register it first; the sweep resolves it.
|
||||
services.AddZbSecrets(config, SecretsSectionPath);
|
||||
|
||||
// Throws when Secrets:GrpcHub:Endpoint or :BearerToken is unset. Same reasoning.
|
||||
services.AddZbSecretsGrpcHubClient(config, GrpcHubSectionPath);
|
||||
break;
|
||||
default:
|
||||
throw new ArgumentOutOfRangeException(nameof(role), role,
|
||||
"Unknown SecretsNodeRole — a new role must decide explicitly which half " +
|
||||
"of the gRPC hub it composes rather than fall into either branch.");
|
||||
}
|
||||
|
||||
return services;
|
||||
}
|
||||
|
||||
// Reached with the mode already validated (UsesGrpcHub resolves it on every path), so a
|
||||
// false answer here means SqlServer mode or replication off — never an unvetted mode value.
|
||||
var enabled = config.GetValue<bool>(ReplicationEnabledKey);
|
||||
var connectionString = config[HubConnectionStringKey];
|
||||
|
||||
if (enabled && !string.IsNullOrWhiteSpace(connectionString))
|
||||
{
|
||||
// Registers the validated hub options, decorates ISecretStore with
|
||||
// ReplicatingSecretStore, and adds the hub-schema migration + bidirectional sweep
|
||||
// hosted services. Calls AddZbSecrets internally — do NOT also call it here.
|
||||
return services.AddZbSecretsSqlServerReplication(
|
||||
// Registers the shared SQL-Server store as ISecretStore directly — no hub, no
|
||||
// follower sweep. Both central nodes just point at the same database, so they serve
|
||||
// identical manifests by construction.
|
||||
return services.AddZbSecretsSqlServerStore(
|
||||
config, SecretsSectionPath, SqlServerSectionPath);
|
||||
}
|
||||
|
||||
if (enabled)
|
||||
{
|
||||
// Asked for replication but gave nothing to replicate against. Falling back to the
|
||||
// local-only store keeps the node up, but it is silently NOT participating in the
|
||||
// cluster, so say so loudly rather than letting it look healthy.
|
||||
Serilog.Log.Warning(
|
||||
"{Key} is true but {ConnKey} is empty — clustered secret replication is DISABLED "
|
||||
+ "and this node's secrets are local-only. The hub connection string cannot come "
|
||||
+ "from the hub itself; supply it via environment variable or seed it in this "
|
||||
+ "node's local store.",
|
||||
ReplicationEnabledKey,
|
||||
HubConnectionStringKey);
|
||||
}
|
||||
|
||||
return services.AddZbSecrets(config, SecretsSectionPath);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Maps the central-hosted secrets hub endpoint, but only on a node whose configuration
|
||||
/// actually registered it. A no-op otherwise.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Central-only by construction: <see cref="AddScadaBridgeSecrets"/> registers the hub service
|
||||
/// and its bearer interceptor solely for <see cref="SecretsNodeRole.Central"/>, and a site node
|
||||
/// never reaches this call. The endpoint lands on the shared endpoint routing, so it is served
|
||||
/// on the dedicated h2c control-plane listener (<c>ScadaBridge:Node:CentralGrpcPort</c>,
|
||||
/// default 8083) — the same listener and the same addressing convention sites already use to
|
||||
/// reach <c>CentralControlService</c>.
|
||||
/// </remarks>
|
||||
/// <param name="endpoints">The endpoint route builder.</param>
|
||||
/// <param name="config">Application configuration.</param>
|
||||
/// <returns>The same <paramref name="endpoints"/> instance, for chaining.</returns>
|
||||
public static IEndpointRouteBuilder MapScadaBridgeSecretsHub(
|
||||
this IEndpointRouteBuilder endpoints,
|
||||
IConfiguration config)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(endpoints);
|
||||
ArgumentNullException.ThrowIfNull(config);
|
||||
|
||||
if (!UsesGrpcHub(config))
|
||||
{
|
||||
return endpoints;
|
||||
}
|
||||
|
||||
return endpoints.MapZbSecretsHub();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -48,14 +48,27 @@ namespace ZB.MOM.WW.ScadaBridge.Host;
|
||||
public static class SiteLocalDbSetup
|
||||
{
|
||||
/// <summary>
|
||||
/// The ten tables that participate in replication — registered when this node has a peer,
|
||||
/// The eleven tables that participate in replication — registered when this node has a peer,
|
||||
/// and deregistered (stale triggers dropped) when it does not.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <c>notification_lists</c> and <c>smtp_configurations</c> are created by the schema but
|
||||
/// deliberately absent here; see the rationale in <see cref="OnReady"/>. The list is shared by
|
||||
/// both branches on purpose: a table added to one and not the other would either register
|
||||
/// without ever being cleaned up, or be cleaned up on a node that never registered it.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <c>deployed_secrets</c> (ciphertext-only, see <c>SecretArtifact</c>) is the eleventh
|
||||
/// entry, added for site-secrets artifact deploy. It has an explicit primary key
|
||||
/// (<c>name</c>) and no FK, so it registers like any other member of this list.
|
||||
/// <c>mirrored_secrets</c> is deliberately NOT here, for the same reason
|
||||
/// <c>notification_lists</c>/<c>smtp_configurations</c> are not: which names THIS node has
|
||||
/// mirrored into its own local secret store is per-node fact, not site-wide state — the
|
||||
/// two site nodes can be at different points in mirroring the same replicated
|
||||
/// <c>deployed_secrets</c> row, and replicating the ledger would make one node's progress
|
||||
/// masquerade as the other's.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
internal static readonly string[] ReplicatedTables =
|
||||
[
|
||||
@@ -69,6 +82,7 @@ public static class SiteLocalDbSetup
|
||||
"database_connections",
|
||||
"data_connection_definitions",
|
||||
"native_alarm_state",
|
||||
"deployed_secrets",
|
||||
];
|
||||
|
||||
/// <summary>
|
||||
@@ -122,11 +136,12 @@ public static class SiteLocalDbSetup
|
||||
//
|
||||
// The PER-TABLE loop is load-bearing as of LocalDb 0.3.0, not just tidy. The snapshot
|
||||
// flag baselining raises is now per-table, and it is raised by the call that actually
|
||||
// seeded something — so adding an ELEVENTH table to ReplicatedTables on an already
|
||||
// replicating site now streams a snapshot of that one table, where through 0.2.x the
|
||||
// flag was a single per-database bit and the same edit re-streamed all eleven in full,
|
||||
// in both directions. Nothing about the first boot changes: enabling replication on a
|
||||
// site that has been running without it seeds all ten at once, the flagged set is
|
||||
// seeded something — so site-secrets artifact deploy adding deployed_secrets as an
|
||||
// ELEVENTH entry to ReplicatedTables on an already-replicating site streams a
|
||||
// snapshot of only that one table, where through 0.2.x the flag was a single
|
||||
// per-database bit and the same edit would have re-streamed all eleven in full, in
|
||||
// both directions. Nothing about the first boot changes: enabling replication on a
|
||||
// site that has been running without it seeds all eleven at once, the flagged set is
|
||||
// therefore every registered table, and the library sends that as an ordinary full
|
||||
// snapshot. A hypothetical "register the whole list in one call" API would silently
|
||||
// give the coarse behaviour back.
|
||||
|
||||
@@ -115,15 +115,16 @@ public static class SiteServiceRegistration
|
||||
services.AddZbLocalDbReplication(config);
|
||||
|
||||
services.AddDataConnectionLayer();
|
||||
// Local SQLite store by default. With Secrets:Replication:Enabled true it becomes either a
|
||||
// local store synced against a shared SQL-Server hub (Mode=SqlServer) or — the production
|
||||
// topology, scadaproj#3 — the FOLLOWER half of the pull-only gRPC secrets hub (Mode=Grpc),
|
||||
// sweeping central over the same gRPC path the control plane already uses instead of
|
||||
// holding a connection string to central's database. A site never hosts the hub; passing
|
||||
// the role explicitly is what makes that a property of this composition root.
|
||||
//
|
||||
// Either way the node keeps a FULL local store, so it resolves secrets straight through a
|
||||
// WAN outage to central and simply converges late. See SecretsRegistration.
|
||||
// A Site node's secret store is ALWAYS the local SQLite store — never a replication
|
||||
// decorator. Task 11 retired both the pull-only gRPC secrets hub (central hosts, sites
|
||||
// sweep) and the shared-SQL-Server replication mode that preceded it. This is deliberate,
|
||||
// not merely simplification: ZB.MOM.WW.ScadaBridge.SiteRuntime.Secrets.DeployedSecretsMirror
|
||||
// deletes a name from the local store directly the moment a deploy artifact stops
|
||||
// referencing it, and a replication-decorated store would have published that delete to
|
||||
// every other node sharing the hub — turning one site's unreferencing of a secret into a
|
||||
// fleet-wide deletion. Sites now receive secrets exclusively via Deploy Artifacts, so the
|
||||
// node keeps a FULL local store and simply picks up whatever the next deploy ships. See
|
||||
// SecretsRegistration.
|
||||
services.AddScadaBridgeSecrets(config, SecretsNodeRole.Site);
|
||||
// Adapter that surfaces the site id to
|
||||
// StoreAndForwardService through DI WITHOUT introducing a
|
||||
|
||||
@@ -26,6 +26,13 @@ public static class StartupValidator
|
||||
|
||||
var seedNodes = configuration.GetSection("ScadaBridge:Cluster:SeedNodes").Get<List<string>>();
|
||||
|
||||
// Secrets replication (the gRPC hub, and the SQL-Server ReplicatingSecretStore mode that
|
||||
// preceded it) is retired in ScadaBridge (Task 11) — sites receive secrets exclusively via
|
||||
// Deploy Artifacts, and a replicated site store would publish DeployedSecretsMirror's local
|
||||
// deletes fleet-wide. Checked unconditionally (both roles): a Site used to be the gRPC
|
||||
// hub's follower half, so the flag was never Central-only.
|
||||
var secretsReplicationEnabled = IsSecretsReplicationEnabled(configuration);
|
||||
|
||||
// GrpcPort: default 8083 when absent; only fails the range rule when the key is
|
||||
// present AND invalid. The out-param assignment mirrors the original so the
|
||||
// resolved grpcPort feeds the cross-field rules even on a parse failure.
|
||||
@@ -49,6 +56,12 @@ public static class StartupValidator
|
||||
.Require("ScadaBridge:Node:RemotingPort",
|
||||
_ => portValid,
|
||||
"must be 1-65535")
|
||||
// Retired for BOTH roles (Task 11) — a Site could set this too, as the gRPC hub's
|
||||
// follower half. Deliberately NOT folded into a role-scoped block below.
|
||||
.Require("Secrets:Replication:Enabled",
|
||||
_ => !secretsReplicationEnabled,
|
||||
"Secrets replication (the gRPC hub) is retired in ScadaBridge — sites receive "
|
||||
+ "secrets with Deploy Artifacts; central uses Secrets:Store:Provider=SqlServer")
|
||||
// SiteId (Site only) — note: OUTSIDE the big Site block in the original,
|
||||
// so it must run before the unconditional SeedNodes-count rule.
|
||||
.When(role == "Site", p => p
|
||||
@@ -97,6 +110,17 @@ public static class StartupValidator
|
||||
value => string.IsNullOrWhiteSpace(value)
|
||||
|| (Enum.TryParse<InboundApiKeyStoreProvider>(value, ignoreCase: true, out var provider)
|
||||
&& Enum.IsDefined(provider)),
|
||||
"must be 'Sqlite' or 'SqlServer' when set")
|
||||
// The central shared secret store provider (Secrets:Store:Provider — a fleet that
|
||||
// ships secrets to sites on every artifact deploy needs a clustered central pair to
|
||||
// see the same secret set). Not required — an unset value defaults to Sqlite,
|
||||
// per-node, mirroring the inbound API-key store default above — but when it IS set,
|
||||
// it must be one of the two legal values so a typo fails fast at boot instead of
|
||||
// silently keeping the per-node SQLite store it looks like it opted out of.
|
||||
.Require("Secrets:Store:Provider",
|
||||
value => string.IsNullOrWhiteSpace(value)
|
||||
|| string.Equals(value, "Sqlite", StringComparison.OrdinalIgnoreCase)
|
||||
|| string.Equals(value, "SqlServer", StringComparison.OrdinalIgnoreCase),
|
||||
"must be 'Sqlite' or 'SqlServer' when set"))
|
||||
// SeedNodes count (unconditional, after SiteId).
|
||||
.Require("ScadaBridge:Cluster:SeedNodes",
|
||||
@@ -226,20 +250,78 @@ public static class StartupValidator
|
||||
var role = configuration.GetSection("ScadaBridge:Node")["Role"];
|
||||
var seedNodes = configuration.GetSection("ScadaBridge:Cluster:SeedNodes").Get<List<string>>();
|
||||
var provider = InboundApiKeyStoreOptions.ReadProvider(configuration);
|
||||
var secretsStoreProvider = configuration["Secrets:Store:Provider"];
|
||||
var secretsReplicationEnabled = IsSecretsReplicationEnabled(configuration);
|
||||
var isClusteredCentral = role == "Central" && seedNodes is { Count: >= 2 };
|
||||
|
||||
if (role == "Central"
|
||||
&& seedNodes is { Count: >= 2 }
|
||||
&& provider == InboundApiKeyStoreProvider.Sqlite)
|
||||
var warnings = new List<string>();
|
||||
|
||||
if (isClusteredCentral && provider == InboundApiKeyStoreProvider.Sqlite)
|
||||
{
|
||||
return new[]
|
||||
{
|
||||
$"Inbound API keys use the per-node SQLite store on a {seedNodes.Count}-node central "
|
||||
warnings.Add(
|
||||
$"Inbound API keys use the per-node SQLite store on a {seedNodes!.Count}-node central "
|
||||
+ "cluster; a failover will reject every inbound client until keys are re-issued. Set "
|
||||
+ "ScadaBridge:InboundApi:ApiKeyStore:Provider=SqlServer (design 2026-09-02)."
|
||||
};
|
||||
+ "ScadaBridge:InboundApi:ApiKeyStore:Provider=SqlServer (design 2026-09-02).");
|
||||
}
|
||||
|
||||
return Array.Empty<string>();
|
||||
// Secrets:Store:Provider defaults to Sqlite (per-node) when unset, mirroring the inbound
|
||||
// API-key store default above — so this fires both when the key is absent and when it is
|
||||
// explicitly "Sqlite". A fleet that ships secrets to sites on every artifact deploy needs
|
||||
// both central nodes to deploy the SAME secret set; a per-node store means each node has
|
||||
// silently drifted the moment anyone edits a secret on only one of them.
|
||||
//
|
||||
// No Secrets:Replication:Enabled gate here any more (Task 11): that flag is now a hard
|
||||
// Validate() failure on either role (the gRPC hub and SQL-Server secrets replication are
|
||||
// both retired), so CollectWarnings never runs with it true on a config that also passed
|
||||
// Validate — the old gate that suppressed this warning when replication was on is dead code
|
||||
// and would otherwise mask the exact per-node-drift risk it exists to catch on any caller
|
||||
// that invokes CollectWarnings directly.
|
||||
if (isClusteredCentral
|
||||
&& !string.Equals(secretsStoreProvider, "SqlServer", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
warnings.Add(
|
||||
"Secrets use a per-node SQLite store on a clustered central — each node would "
|
||||
+ "deploy its own secret set to sites; set Secrets:Store:Provider=SqlServer");
|
||||
}
|
||||
|
||||
// Review round 1, Minor #4: a config that still carries Secrets:Replication or
|
||||
// Secrets:GrpcHub child keys — left over from before Task 11 retired the hub — is not
|
||||
// itself an error (Validate already rejects Enabled=true outright), but every key under
|
||||
// either section is now silently ignored. That is worth a Warning so an operator cleans
|
||||
// QA/prod appsettings rather than assuming a still-present Mode/BearerToken/Endpoint value
|
||||
// does anything. Excludes the bare "Enabled" key itself: "Enabled: false" alone is the
|
||||
// normal, still-valid shape of every existing deployment's config and must not warn.
|
||||
var hasStaleReplicationKeys = configuration.GetSection("Secrets:Replication").GetChildren()
|
||||
.Any(child => !string.Equals(child.Key, "Enabled", StringComparison.OrdinalIgnoreCase));
|
||||
var hasStaleGrpcHubKeys = configuration.GetSection("Secrets:GrpcHub").GetChildren().Any();
|
||||
|
||||
if (!secretsReplicationEnabled && (hasStaleReplicationKeys || hasStaleGrpcHubKeys))
|
||||
{
|
||||
warnings.Add(
|
||||
"Secrets:Replication / Secrets:GrpcHub settings are ignored — the secrets hub is "
|
||||
+ "retired; remove them");
|
||||
}
|
||||
|
||||
return warnings;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Whether <c>Secrets:Replication:Enabled</c> is "set" — read as a raw string rather than
|
||||
/// <c>IConfiguration.GetValue<bool></c> (review round 1, Minor #3), which throws a raw
|
||||
/// binder <see cref="InvalidOperationException"/> on a non-boolean value like <c>"yes"</c> or
|
||||
/// <c>"1"</c>, so a malformed leftover key would surface as an unrelated parse crash instead of
|
||||
/// the designed retirement message in <see cref="Validate"/>. Any value other than absent,
|
||||
/// empty, or (case-insensitively) <c>"false"</c> counts as "set" — deliberately stricter than
|
||||
/// bool parsing, so a typo'd leftover (<c>Enabled: 1</c>, <c>Enabled: yes</c>) still gets the
|
||||
/// explanatory message rather than silently passing as falsy. Shared by <see cref="Validate"/>
|
||||
/// (which rejects "set" outright) and <see cref="CollectWarnings"/> (which, for the "not set"
|
||||
/// case, separately warns about any OTHER leftover key under the same retired sections).
|
||||
/// </summary>
|
||||
private static bool IsSecretsReplicationEnabled(IConfiguration configuration)
|
||||
{
|
||||
var raw = configuration["Secrets:Replication:Enabled"];
|
||||
return !string.IsNullOrEmpty(raw)
|
||||
&& !string.Equals(raw, "false", StringComparison.OrdinalIgnoreCase);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
|
||||
@@ -41,11 +41,11 @@
|
||||
<PackageReference Include="ZB.MOM.WW.Secrets" />
|
||||
<PackageReference Include="ZB.MOM.WW.Secrets.Abstractions" />
|
||||
<PackageReference Include="ZB.MOM.WW.Secrets.Ui" />
|
||||
<!-- Secrets:Store:Provider=SqlServer only (Task 5): points a clustered central pair at one
|
||||
shared SQL-Server secret store directly, no hub. The gRPC secrets hub package this project
|
||||
used to also reference (ZB.MOM.WW.Secrets.Replicator.Grpc) was retired in Task 11 — see
|
||||
SecretsRegistration. -->
|
||||
<PackageReference Include="ZB.MOM.WW.Secrets.Replicator.SqlServer" />
|
||||
<!-- Both halves of the pull-only central secrets hub ship in one package: central
|
||||
hosts the gRPC service, a site runs the sweep client. Which half a node composes
|
||||
is decided at registration time by its role — see SecretsRegistration. -->
|
||||
<PackageReference Include="ZB.MOM.WW.Secrets.Replicator.Grpc" />
|
||||
<PackageReference Include="ZB.MOM.WW.LocalDb" />
|
||||
<PackageReference Include="ZB.MOM.WW.LocalDb.Replication" />
|
||||
</ItemGroup>
|
||||
|
||||
@@ -14,30 +14,15 @@
|
||||
"MasterKey": { "Source": "Environment", "EnvVarName": "ZB_SECRETS_MASTER_KEY" },
|
||||
"RunMigrationsOnStartup": true,
|
||||
"ResolveCacheTtl": "00:00:30",
|
||||
"Replication": {
|
||||
"_comment": "Default-OFF pin: no deployment enables clustered secret replication by shipping this file. Mode selects the transport when Enabled is true - 'SqlServer' (the default, kept so an existing config that sets only Enabled behaves exactly as before) or 'Grpc' (the PRODUCTION topology, scadaproj#3: central hosts a pull-only hub under Secrets:GrpcHub and sites sweep it, so a site never needs a connection string to central's database). Every node in either topology must carry the SAME ZB_SECRETS_MASTER_KEY - only ciphertext crosses the wire, so a node with a different KEK fails closed on resolve with a kek_id mismatch that reads like corruption but is a deployment error.",
|
||||
"Enabled": false,
|
||||
"Mode": "SqlServer"
|
||||
"Store": {
|
||||
"_comment": "CENTRAL ONLY (a Site always ignores this and keeps its local SQLite store, permanently - Task 11 retired secrets replication, so there is no decorator left for a site to be given). 'Sqlite' (default) is per-node: on a clustered central, each node's secrets can silently drift the moment anyone edits one on only a single node, which then ships an inconsistent secret set to sites on artifact deploy - StartupValidator warns on a >=2-seed central left on Sqlite. 'SqlServer' points BOTH central nodes at ONE shared SQL-Server database directly (Secrets:SqlServer:ConnectionString, fail-closed on a blank or ${secret:} value), so the pair can never diverge.",
|
||||
"Provider": "Sqlite"
|
||||
},
|
||||
"_replication": "Secrets replication (the gRPC hub, and the SQL-Server ReplicatingSecretStore mode that preceded it) is retired - Task 11. Secrets:Replication:Enabled=true is now a hard startup failure on either role; sites receive secrets exclusively via Deploy Artifacts, and Secrets:Store:Provider=SqlServer (above) is how a clustered central shares one secret set.",
|
||||
"SqlServer": {
|
||||
"ConnectionString": "",
|
||||
"SchemaName": "zbsecrets",
|
||||
"SyncInterval": "00:00:30",
|
||||
"SyncOnStartup": true,
|
||||
"_comment": "Two modes read this section. Mode=SqlServer: the hub connstr must be seeded in the LOCAL store (or supplied via env) - it cannot come from the hub itself. Mode=Grpc: REQUIRED on CENTRAL, where it points BOTH central nodes at ONE shared SQL-Server secret store so the pair cannot diverge (scadaproj#4 - an independent store per central node let one hub serve an authenticated EMPTY manifest); it must be a LITERAL or environment value (Secrets__SqlServer__ConnectionString), never a ${secret:} reference - the pre-host expander needs this store to resolve references, so the reference could never resolve (bootstrap circularity; registration rejects it). SITES leave it empty in every mode - sites talk to central, never to central's database."
|
||||
},
|
||||
"GrpcHub": {
|
||||
"_comment": "Read ONLY when Secrets:Replication:Enabled is true AND Mode=Grpc; inert otherwise. ONE section, both halves: CENTRAL reads BearerToken + MaxNamesPerRequest and hosts the hub on its CentralGrpcPort h2c listener (default 8083, alongside CentralControlService); a SITE reads Endpoint + BearerToken + the sweep timings and pulls. Replication is pull-only by wire contract - the proto has no write RPC - so secrets originate at central and a site cannot push. FAIL-CLOSED: unlike SqlServer mode there is no local-only fallback; a missing BearerToken (either role) or Endpoint (site) is a startup failure naming the key.",
|
||||
"_bearerToken": "Shared credential every follower presents. Supply it from appsettings or the environment (Secrets__GrpcHub__BearerToken), NOT as a ${secret:...} reference - resolving that reference is what the hub exists to make possible, so it cannot come from the hub. Same rationale and same handling as the mesh pre-shared keys. Never commit a real value here; the empty default below is fail-closed, not open.",
|
||||
"BearerToken": "",
|
||||
"_endpoint": "SITE ONLY. Absolute http/https URI of a CENTRAL node's gRPC (h2c) port - the same address family as ScadaBridge:Communication:CentralGrpcEndpoints, e.g. 'http://central-a-host:8083'. NOT via Traefik (HTTP/1 only). With FallbackEndpoints left empty the hub client dials this SINGLE endpoint only: a sweep against a downed central-a then stalls rather than failing over - survivable because the sweep is best-effort (one warning per interval, then retry) and the site keeps serving its full local last-known-good store, but secrets stop converging until that central node returns. Listing the second central under FallbackEndpoints removes that stall.",
|
||||
"Endpoint": "",
|
||||
"_fallbackEndpoints": "SITE ONLY. List the SECOND central node's hub endpoint here (same URI form as Endpoint, e.g. 'http://central-b-host:8083') so a sweep fails over across the central pair instead of stalling on a downed primary. Safe ONLY because both central nodes serve ONE shared SQL secret store, so either hub answers with the same manifest - NEVER list an endpoint backed by an independent store: failing over to an emptier hub is a silent convergence stop, the exact defect scadaproj#4 recorded. Empty by default; leaving it empty keeps the pre-0.5.0 single-endpoint behavior byte-identical.",
|
||||
"FallbackEndpoints": [],
|
||||
"SyncInterval": "00:00:30",
|
||||
"SyncOnStartup": true,
|
||||
"CallDeadline": "00:00:30",
|
||||
"MaxNamesPerRequest": 1000
|
||||
"_comment": "Read ONLY when Secrets:Store:Provider=SqlServer: points BOTH central nodes at ONE shared SQL-Server secret store so the pair cannot diverge. Must be a LITERAL or environment value (Secrets__SqlServer__ConnectionString), never a ${secret:} reference - the pre-host expander needs this store to resolve references, so the reference could never resolve (bootstrap circularity; registration rejects it). SITES leave it empty - sites talk to central, never to central's database. Every node that handles deployed secrets must still carry the SAME ZB_SECRETS_MASTER_KEY - only ciphertext crosses the wire, so a node with a different KEK fails closed on resolve with a kek_id mismatch that reads like corruption but is a deployment error."
|
||||
}
|
||||
},
|
||||
"Serilog": {
|
||||
|
||||
@@ -18,6 +18,7 @@ using ZB.MOM.WW.ScadaBridge.SiteRuntime.Messages;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Repositories;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Secrets;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Streaming;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
|
||||
@@ -49,6 +50,15 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
|
||||
/// external-system caching — the invalidation call is then simply skipped.
|
||||
/// </summary>
|
||||
private readonly ExternalSystemDefinitionCache? _externalSystemCache;
|
||||
/// <summary>
|
||||
/// Task 10 (site-secrets artifact deploy): per-node mirror reconciled synchronously by
|
||||
/// <see cref="HandleDeployArtifacts"/> right after the shipped secrets are written to
|
||||
/// <c>deployed_secrets</c>, so the reply's <c>SecretsApplied</c> reflects this node's own
|
||||
/// apply/verify outcome rather than the next periodic pass. Optional/null on nodes/tests
|
||||
/// that never register secret deployment — a shipped <c>command.Secrets</c> with no mirror
|
||||
/// registered fails the deploy (see <see cref="HandleDeployArtifacts"/>).
|
||||
/// </summary>
|
||||
private readonly DeployedSecretsMirror? _secretsMirror;
|
||||
private readonly SiteStreamManager? _streamManager;
|
||||
private readonly SiteRuntimeOptions _options;
|
||||
private readonly ILogger<DeploymentManagerActor> _logger;
|
||||
@@ -306,6 +316,12 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
|
||||
/// ensure-started heal uses; defaults to reading from <paramref name="storage"/>. Primarily
|
||||
/// for tests that need to hold the read open mid-flight to drive the row-101 race.
|
||||
/// </param>
|
||||
/// <param name="secretsMirror">
|
||||
/// Task 10 (site-secrets artifact deploy): this node's <see cref="DeployedSecretsMirror"/>,
|
||||
/// reconciled synchronously by <see cref="HandleDeployArtifacts"/> right after a shipped
|
||||
/// <c>command.Secrets</c> set is written to <c>deployed_secrets</c>. Optional/null on
|
||||
/// nodes/tests that never register secret deployment.
|
||||
/// </param>
|
||||
public DeploymentManagerActor(
|
||||
SiteStorageService storage,
|
||||
ScriptCompilationService compilationService,
|
||||
@@ -323,9 +339,11 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
|
||||
ExternalSystemDefinitionCache? externalSystemCache = null,
|
||||
ScriptExecutionScheduler? scriptScheduler = null,
|
||||
Func<Task<List<StoredDataConnectionDefinition>>>? connectionDefinitionLoader = null,
|
||||
Func<string, Task<DeployedInstance?>>? deployedConfigLoader = null)
|
||||
Func<string, Task<DeployedInstance?>>? deployedConfigLoader = null,
|
||||
DeployedSecretsMirror? secretsMirror = null)
|
||||
{
|
||||
_scriptScheduler = scriptScheduler;
|
||||
_secretsMirror = secretsMirror;
|
||||
_storage = storage;
|
||||
_compilationService = compilationService;
|
||||
_deployCompileValidator = new DeployCompileValidator(compilationService);
|
||||
@@ -3213,6 +3231,13 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
|
||||
|
||||
Task.Run(async () =>
|
||||
{
|
||||
// M2 (final review): the DCL push for this deploy's data connections is captured
|
||||
// here and sent only AFTER the shipped secrets are written and mirrored (in the
|
||||
// finally below), so a connection whose config references a secret this same deploy
|
||||
// ships resolves it on its FIRST connect instead of failing "not found" and waiting
|
||||
// out the reconnect interval. The definitions are still stored in SQLite at the same
|
||||
// point as before, and the push still goes out if the secrets step throws.
|
||||
ApplyArtifactDataConnectionsToDcl? pendingDclApply = null;
|
||||
try
|
||||
{
|
||||
// Each non-null artifact list is the COMPLETE system-wide set
|
||||
@@ -3310,23 +3335,105 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
|
||||
// artifact bundle. Reconciled removals ride the same
|
||||
// message so the live DCL connection actor is stopped
|
||||
// on the actor thread alongside the hash-cache eviction.
|
||||
self.Tell(new ApplyArtifactDataConnectionsToDcl(
|
||||
command.DataConnections, removedConnections));
|
||||
// Sent after the secrets step — see pendingDclApply above (M2).
|
||||
pendingDclApply = new ApplyArtifactDataConnectionsToDcl(
|
||||
command.DataConnections, removedConnections);
|
||||
}
|
||||
|
||||
// SMTP configuration is
|
||||
// central-only and is never stored on a site (see the purge above).
|
||||
|
||||
// Site-secrets artifact deploy (Task 10): write the shipped secrets to the
|
||||
// replicated deployed_secrets table, then reconcile THIS node's own mirror
|
||||
// synchronously — the reply's SecretsApplied must reflect this node's own
|
||||
// apply/verify outcome, not whatever the next periodic pass finds. A null
|
||||
// command.Secrets means "central didn't ship secrets" (old-central
|
||||
// compatible) and touches nothing; a non-null Secrets with no mirror
|
||||
// registered on this node fails the deploy rather than silently applying
|
||||
// nothing.
|
||||
int? secretsApplied = null;
|
||||
string? secretError = null;
|
||||
if (command.Secrets != null)
|
||||
{
|
||||
if (_secretsMirror == null)
|
||||
{
|
||||
if (command.Secrets.Count > 0)
|
||||
{
|
||||
secretError = "Secret deployment is not enabled on this site node";
|
||||
}
|
||||
else
|
||||
{
|
||||
secretsApplied = 0;
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
_logger.LogInformation(
|
||||
"Applying {Count} deployed secret(s), deploymentId={DeploymentId}",
|
||||
command.Secrets.Count, command.DeploymentId);
|
||||
await _storage.UpsertDeployedSecretsAsync(command.Secrets, command.DeploymentId);
|
||||
var mirror = await _secretsMirror.ReconcileAsync(CancellationToken.None);
|
||||
|
||||
// The mirror reconciles the WHOLE site-wide deployed_secrets table
|
||||
// every pass — not only the names THIS deploy just shipped — so a
|
||||
// failure unrelated to this deploy's own secrets (another instance's
|
||||
// already-deployed secret still failing, or the removal of a
|
||||
// no-longer-referenced one) must never fail THIS response or show up
|
||||
// in its ErrorMessage. Only failures among the SHIPPED names count.
|
||||
var shippedNames = new HashSet<string>(
|
||||
command.Secrets.Select(s => s.Name), StringComparer.Ordinal);
|
||||
var shippedFailureMessages = new List<string>();
|
||||
var shippedFailedNames = new HashSet<string>(StringComparer.Ordinal);
|
||||
|
||||
for (var i = 0; i < mirror.FailedNames.Count; i++)
|
||||
{
|
||||
var failedName = mirror.FailedNames[i];
|
||||
var failedMessage = mirror.Failed[i];
|
||||
if (shippedNames.Contains(failedName))
|
||||
{
|
||||
shippedFailedNames.Add(failedName);
|
||||
shippedFailureMessages.Add(failedMessage);
|
||||
}
|
||||
else
|
||||
{
|
||||
_logger.LogWarning(
|
||||
"Deployed secrets mirror reported a failure unrelated to " +
|
||||
"this deploy ({DeploymentId}): {Failure}",
|
||||
command.DeploymentId, failedMessage);
|
||||
}
|
||||
}
|
||||
|
||||
// Never negative (ruling): shippedFailedNames is a set intersected
|
||||
// against the shipped names, so this can only go below zero if the
|
||||
// shipped list itself contains duplicate names.
|
||||
secretsApplied = Math.Max(0, command.Secrets.Count - shippedFailedNames.Count);
|
||||
if (shippedFailureMessages.Count > 0)
|
||||
{
|
||||
secretError = string.Join("; ", shippedFailureMessages);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Replicate artifacts to standby node
|
||||
|
||||
return new ArtifactDeploymentResponse(
|
||||
command.DeploymentId, "", true, null, DateTimeOffset.UtcNow);
|
||||
command.DeploymentId, "", secretError is null, secretError, DateTimeOffset.UtcNow,
|
||||
secretsApplied);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
return new ArtifactDeploymentResponse(
|
||||
command.DeploymentId, "", false, ex.Message, DateTimeOffset.UtcNow);
|
||||
}
|
||||
finally
|
||||
{
|
||||
// Runs before the Task completes, so the DCL push is still enqueued ahead of the
|
||||
// PipeTo reply, as it was before the reorder.
|
||||
if (pendingDclApply != null)
|
||||
{
|
||||
self.Tell(pendingDclApply);
|
||||
}
|
||||
}
|
||||
}).PipeTo(sender);
|
||||
}
|
||||
|
||||
|
||||
@@ -3,7 +3,8 @@ using Microsoft.Data.Sqlite;
|
||||
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
|
||||
|
||||
/// <summary>
|
||||
/// DDL for the site's nine configuration tables, extracted from
|
||||
/// DDL for the site's eleven configuration tables (nine original plus
|
||||
/// <c>deployed_secrets</c> and <c>mirrored_secrets</c>), extracted from
|
||||
/// <see cref="SiteStorageService"/> so it can be applied by whoever owns the database
|
||||
/// file.
|
||||
/// <para>
|
||||
@@ -22,7 +23,7 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
|
||||
public static class SiteStorageSchema
|
||||
{
|
||||
/// <summary>
|
||||
/// Creates the nine site configuration tables when absent, and additively upgrades
|
||||
/// Creates the eleven site configuration tables when absent, and additively upgrades
|
||||
/// tables created by an older build. Idempotent — safe to run on every startup.
|
||||
/// </summary>
|
||||
/// <param name="connection">An open connection to the database that should hold the tables.</param>
|
||||
@@ -111,6 +112,30 @@ public static class SiteStorageSchema
|
||||
metadata_json TEXT,
|
||||
PRIMARY KEY (instance_unique_name, source_canonical_name, source_reference)
|
||||
);
|
||||
|
||||
-- Deployed secrets (ciphertext only — see SecretArtifact). Replicated between the
|
||||
-- two site nodes by LocalDb CDC (SiteLocalDbSetup.ReplicatedTables). A full-set
|
||||
-- upsert soft-deletes rows absent from the incoming set via is_removed, never a
|
||||
-- mass DELETE on a replicated table (that would ship as a destructive resync).
|
||||
CREATE TABLE IF NOT EXISTS deployed_secrets (
|
||||
name TEXT PRIMARY KEY,
|
||||
row_json TEXT NOT NULL,
|
||||
revision INTEGER NOT NULL,
|
||||
updated_utc TEXT NOT NULL,
|
||||
is_removed INTEGER NOT NULL DEFAULT 0,
|
||||
deployment_id TEXT NOT NULL,
|
||||
applied_at TEXT NOT NULL
|
||||
);
|
||||
|
||||
-- Node-local ledger of which secret names THIS node has mirrored into its own
|
||||
-- secret store. Deliberately NOT in SiteLocalDbSetup.ReplicatedTables: mirroring
|
||||
-- is a per-node action against a per-node secret store, so which names are
|
||||
-- mirrored (and at what revision) is node-specific, not site-wide state.
|
||||
CREATE TABLE IF NOT EXISTS mirrored_secrets (
|
||||
name TEXT PRIMARY KEY,
|
||||
revision INTEGER NOT NULL,
|
||||
mirrored_at TEXT NOT NULL
|
||||
);
|
||||
";
|
||||
command.ExecuteNonQuery();
|
||||
}
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
using System.Text.Json;
|
||||
using Microsoft.Data.Sqlite;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using ZB.MOM.WW.LocalDb;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
|
||||
|
||||
@@ -658,6 +660,270 @@ public class SiteStorageService
|
||||
_logger.LogDebug("Cleared native alarm state for {Instance}", instanceName);
|
||||
}
|
||||
|
||||
// ── Deployed Secrets store (LocalDb-replicated ciphertext) ──
|
||||
|
||||
/// <summary>
|
||||
/// Full-set upsert of the site's deployed secrets: every <see cref="SecretArtifact"/> in
|
||||
/// <paramref name="secrets"/> is written (<c>is_removed = 0</c>), and every row not named
|
||||
/// in the set is soft-deleted (<c>is_removed = 1</c>) rather than actually removed.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// Central ships the complete set of secrets referenced by a site's artifacts on every
|
||||
/// artifact deploy, mirroring the reconcile contract of
|
||||
/// <see cref="DeleteRowsExceptAsync"/> for the other artifact tables. Unlike those tables,
|
||||
/// <c>deployed_secrets</c> is never hard-deleted: it is one of
|
||||
/// <c>SiteLocalDbSetup.ReplicatedTables</c>, and a mass <c>DELETE</c> on a replicated table
|
||||
/// would be captured and shipped to the peer as a destructive resync — the same hazard
|
||||
/// <c>StoreAndForwardStorage.ReplaceAllAsync</c> was deleted for in LocalDb Phase 2. A row
|
||||
/// that a later deploy re-adds simply flips back to <c>is_removed = 0</c>.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <c>row_json</c> is <c>JsonSerializer.Serialize(secret)</c> — the <see cref="SecretArtifact"/>
|
||||
/// record verbatim, so its <c>byte[]</c> ciphertext fields serialize as base64 and nothing
|
||||
/// but ciphertext ever lands in this column.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
/// <param name="secrets">The complete set of secrets that should exist on this site.</param>
|
||||
/// <param name="deploymentId">The deployment identifier that applied this set.</param>
|
||||
/// <returns>
|
||||
/// A task that resolves to the names that were applied (inserted or updated) and the
|
||||
/// names that were newly marked removed by this call (rows that were previously
|
||||
/// <c>is_removed = 0</c>; a name already removed by an earlier call is not repeated here).
|
||||
/// </returns>
|
||||
public async Task<(IReadOnlyList<string> Applied, IReadOnlyList<string> Removed)> UpsertDeployedSecretsAsync(
|
||||
IReadOnlyList<SecretArtifact> secrets, string deploymentId)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(secrets);
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(deploymentId);
|
||||
|
||||
await using var connection = OpenConnection();
|
||||
await using var transaction = (SqliteTransaction)await connection.BeginTransactionAsync();
|
||||
|
||||
var appliedAt = DateTimeOffset.UtcNow.ToString("O");
|
||||
var applied = new List<string>(secrets.Count);
|
||||
|
||||
await using (var upsert = connection.CreateCommand())
|
||||
{
|
||||
upsert.Transaction = transaction;
|
||||
upsert.CommandText = @"
|
||||
INSERT INTO deployed_secrets (name, row_json, revision, updated_utc, is_removed, deployment_id, applied_at)
|
||||
VALUES (@name, @json, @revision, @updatedUtc, 0, @deploymentId, @appliedAt)
|
||||
ON CONFLICT(name) DO UPDATE SET
|
||||
row_json = excluded.row_json,
|
||||
revision = excluded.revision,
|
||||
updated_utc = excluded.updated_utc,
|
||||
is_removed = 0,
|
||||
deployment_id = excluded.deployment_id,
|
||||
applied_at = excluded.applied_at";
|
||||
|
||||
var pName = upsert.Parameters.Add("@name", SqliteType.Text);
|
||||
var pJson = upsert.Parameters.Add("@json", SqliteType.Text);
|
||||
var pRevision = upsert.Parameters.Add("@revision", SqliteType.Integer);
|
||||
var pUpdatedUtc = upsert.Parameters.Add("@updatedUtc", SqliteType.Text);
|
||||
var pDeploymentId = upsert.Parameters.Add("@deploymentId", SqliteType.Text);
|
||||
var pAppliedAt = upsert.Parameters.Add("@appliedAt", SqliteType.Text);
|
||||
pDeploymentId.Value = deploymentId;
|
||||
pAppliedAt.Value = appliedAt;
|
||||
|
||||
foreach (var secret in secrets)
|
||||
{
|
||||
pName.Value = secret.Name;
|
||||
pJson.Value = JsonSerializer.Serialize(secret);
|
||||
pRevision.Value = secret.Revision;
|
||||
pUpdatedUtc.Value = secret.UpdatedUtc.ToString("O");
|
||||
await upsert.ExecuteNonQueryAsync();
|
||||
applied.Add(secret.Name);
|
||||
}
|
||||
}
|
||||
|
||||
// Names present in the incoming set are never candidates for removal, including
|
||||
// one that was previously soft-deleted and has just been re-applied above.
|
||||
var keepParams = secrets.Select((s, i) => (Key: $"@k{i}", Value: s.Name)).ToList();
|
||||
var predicate = keepParams.Count == 0
|
||||
? " WHERE is_removed = 0"
|
||||
: $" WHERE is_removed = 0 AND name NOT IN ({string.Join(", ", keepParams.Select(p => p.Key))})";
|
||||
|
||||
var removed = new List<string>();
|
||||
await using (var select = connection.CreateCommand())
|
||||
{
|
||||
select.Transaction = transaction;
|
||||
select.CommandText = $"SELECT name FROM deployed_secrets{predicate}";
|
||||
foreach (var (key, value) in keepParams)
|
||||
select.Parameters.AddWithValue(key, value);
|
||||
|
||||
await using var reader = await select.ExecuteReaderAsync();
|
||||
while (await reader.ReadAsync())
|
||||
removed.Add(reader.GetString(0));
|
||||
}
|
||||
|
||||
if (removed.Count > 0)
|
||||
{
|
||||
await using var markRemoved = connection.CreateCommand();
|
||||
markRemoved.Transaction = transaction;
|
||||
markRemoved.CommandText = $"UPDATE deployed_secrets SET is_removed = 1{predicate}";
|
||||
foreach (var (key, value) in keepParams)
|
||||
markRemoved.Parameters.AddWithValue(key, value);
|
||||
|
||||
await markRemoved.ExecuteNonQueryAsync();
|
||||
}
|
||||
|
||||
await transaction.CommitAsync();
|
||||
|
||||
_logger.LogInformation(
|
||||
"Deployed secrets reconcile for deployment {DeploymentId}: applied {AppliedCount}, newly removed {RemovedCount}",
|
||||
deploymentId, applied.Count, removed.Count);
|
||||
|
||||
return (applied, removed);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Returns every row in <c>deployed_secrets</c>, including soft-deleted (<c>is_removed</c>)
|
||||
/// ones, with each row's <see cref="SecretArtifact"/> round-tripped from <c>row_json</c>.
|
||||
/// Strict: a row whose <c>row_json</c> cannot be parsed throws. The per-node mirror reads
|
||||
/// through <see cref="GetDeployedSecretsTolerantAsync"/> instead, so one unreadable row is
|
||||
/// reported against its own name rather than aborting the whole reconcile pass.
|
||||
/// </summary>
|
||||
/// <returns>A task that resolves to every stored deployed-secret row.</returns>
|
||||
public async Task<IReadOnlyList<DeployedSecretRow>> GetDeployedSecretsAsync()
|
||||
{
|
||||
var (rows, unreadable) = await GetDeployedSecretsTolerantAsync();
|
||||
if (unreadable.Count > 0)
|
||||
{
|
||||
throw new InvalidOperationException(
|
||||
$"deployed_secrets row '{unreadable[0].Name}' has invalid row_json ({unreadable[0].Error})");
|
||||
}
|
||||
|
||||
return rows;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Returns every row in <c>deployed_secrets</c> like <see cref="GetDeployedSecretsAsync"/>,
|
||||
/// but never throws for a single bad row: a row whose <c>row_json</c> does not deserialize
|
||||
/// is returned in <c>Unreadable</c> (name, removal flag and the parse error — never the
|
||||
/// row's content) instead of failing the whole read (final review M4, per-row isolation).
|
||||
/// </summary>
|
||||
/// <returns>The parsed rows and the rows that could not be parsed.</returns>
|
||||
public async Task<(IReadOnlyList<DeployedSecretRow> Rows, IReadOnlyList<UnreadableDeployedSecretRow> Unreadable)>
|
||||
GetDeployedSecretsTolerantAsync()
|
||||
{
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var command = connection.CreateCommand();
|
||||
command.CommandText = "SELECT name, row_json, is_removed, deployment_id FROM deployed_secrets";
|
||||
|
||||
var results = new List<DeployedSecretRow>();
|
||||
var unreadable = new List<UnreadableDeployedSecretRow>();
|
||||
await using var reader = await command.ExecuteReaderAsync();
|
||||
while (await reader.ReadAsync())
|
||||
{
|
||||
var name = reader.GetString(0);
|
||||
var isRemoved = reader.GetInt64(2) != 0;
|
||||
|
||||
SecretArtifact? secret;
|
||||
string? error = null;
|
||||
try
|
||||
{
|
||||
secret = JsonSerializer.Deserialize<SecretArtifact>(reader.GetString(1));
|
||||
if (secret is null)
|
||||
{
|
||||
error = "row_json deserialized to null";
|
||||
}
|
||||
}
|
||||
catch (Exception ex) when (ex is JsonException or NotSupportedException or InvalidCastException)
|
||||
{
|
||||
secret = null;
|
||||
error = $"{ex.GetType().Name}: {ex.Message}";
|
||||
}
|
||||
|
||||
if (secret is null)
|
||||
{
|
||||
unreadable.Add(new UnreadableDeployedSecretRow(name, isRemoved, error!));
|
||||
continue;
|
||||
}
|
||||
|
||||
results.Add(new DeployedSecretRow(name, secret, isRemoved, reader.GetString(3)));
|
||||
}
|
||||
|
||||
return (results, unreadable);
|
||||
}
|
||||
|
||||
// ── Mirrored Secrets ledger (node-local only — deliberately NOT LocalDb-replicated) ──
|
||||
|
||||
/// <summary>
|
||||
/// Returns every name this node has recorded as mirrored into its own secret store.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <c>mirrored_secrets</c> is not in <c>SiteLocalDbSetup.ReplicatedTables</c> on purpose:
|
||||
/// mirroring writes a name into THIS node's own local secret store, so which names are
|
||||
/// mirrored — and at what revision — is per-node fact, not site-wide state. Replicating it
|
||||
/// would tell this node's ledger about the OTHER node's mirroring progress, which is not
|
||||
/// what it records.
|
||||
/// </remarks>
|
||||
/// <returns>A task that resolves to every mirrored-secret ledger row.</returns>
|
||||
public async Task<IReadOnlyList<MirroredSecretRow>> GetMirroredSecretsAsync()
|
||||
{
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var command = connection.CreateCommand();
|
||||
command.CommandText = "SELECT name, revision, mirrored_at FROM mirrored_secrets";
|
||||
|
||||
var results = new List<MirroredSecretRow>();
|
||||
await using var reader = await command.ExecuteReaderAsync();
|
||||
while (await reader.ReadAsync())
|
||||
{
|
||||
results.Add(new MirroredSecretRow(
|
||||
reader.GetString(0),
|
||||
reader.GetInt64(1),
|
||||
DateTimeOffset.Parse(reader.GetString(2), null, System.Globalization.DateTimeStyles.RoundtripKind)));
|
||||
}
|
||||
|
||||
return results;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Records that this node has mirrored <paramref name="name"/> at <paramref name="revision"/>
|
||||
/// into its own secret store. Uses UPSERT semantics.
|
||||
/// </summary>
|
||||
/// <param name="name">The mirrored secret's name.</param>
|
||||
/// <param name="revision">The revision that was mirrored.</param>
|
||||
/// <returns>A task that completes when the ledger row has been stored or updated.</returns>
|
||||
public async Task SetMirroredSecretAsync(string name, long revision)
|
||||
{
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var command = connection.CreateCommand();
|
||||
command.CommandText = @"
|
||||
INSERT INTO mirrored_secrets (name, revision, mirrored_at)
|
||||
VALUES (@name, @revision, @mirroredAt)
|
||||
ON CONFLICT(name) DO UPDATE SET
|
||||
revision = excluded.revision,
|
||||
mirrored_at = excluded.mirrored_at";
|
||||
|
||||
command.Parameters.AddWithValue("@name", name);
|
||||
command.Parameters.AddWithValue("@revision", revision);
|
||||
command.Parameters.AddWithValue("@mirroredAt", DateTimeOffset.UtcNow.ToString("O"));
|
||||
|
||||
await command.ExecuteNonQueryAsync();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Removes the mirrored-secret ledger row for <paramref name="name"/> (e.g. once the
|
||||
/// secret is no longer deployed to this site and this node has un-mirrored it).
|
||||
/// </summary>
|
||||
/// <param name="name">The mirrored secret's name.</param>
|
||||
/// <returns>A task that completes when the ledger row has been deleted.</returns>
|
||||
public async Task RemoveMirroredSecretAsync(string name)
|
||||
{
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var command = connection.CreateCommand();
|
||||
command.CommandText = "DELETE FROM mirrored_secrets WHERE name = @name";
|
||||
command.Parameters.AddWithValue("@name", name);
|
||||
|
||||
await command.ExecuteNonQueryAsync();
|
||||
}
|
||||
|
||||
// ── Shared Script CRUD ──
|
||||
|
||||
/// <summary>
|
||||
@@ -1095,3 +1361,26 @@ public class StoredDataConnectionDefinition
|
||||
/// </summary>
|
||||
public record NativeAlarmRow(
|
||||
string SourceReference, string ConditionJson, DateTimeOffset LastTransitionAt, string? MetadataJson = null);
|
||||
|
||||
/// <summary>
|
||||
/// A single row from the site-local, LocalDb-replicated <c>deployed_secrets</c> table.
|
||||
/// <paramref name="Secret"/> is the <see cref="SecretArtifact"/> round-tripped from
|
||||
/// <c>row_json</c> — ciphertext only, never plaintext. <paramref name="IsRemoved"/> is true
|
||||
/// for a soft-deleted row (see <see cref="SiteStorageService.UpsertDeployedSecretsAsync"/>);
|
||||
/// the row is retained, not hard-deleted, because this table replicates.
|
||||
/// </summary>
|
||||
public record DeployedSecretRow(string Name, SecretArtifact Secret, bool IsRemoved, string DeploymentId);
|
||||
|
||||
/// <summary>
|
||||
/// A <c>deployed_secrets</c> row whose <c>row_json</c> could not be parsed — returned by
|
||||
/// <see cref="SiteStorageService.GetDeployedSecretsTolerantAsync"/> so the mirror can report it
|
||||
/// against its own name instead of aborting the pass. <paramref name="Error"/> is the parse error
|
||||
/// (exception type and message), never the row's content.
|
||||
/// </summary>
|
||||
public record UnreadableDeployedSecretRow(string Name, bool IsRemoved, string Error);
|
||||
|
||||
/// <summary>
|
||||
/// A single row from the site-local, node-local (NOT LocalDb-replicated) <c>mirrored_secrets</c>
|
||||
/// ledger — the names this node has mirrored into its own secret store, and at what revision.
|
||||
/// </summary>
|
||||
public record MirroredSecretRow(string Name, long Revision, DateTimeOffset MirroredAt);
|
||||
|
||||
@@ -0,0 +1,621 @@
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Secrets;
|
||||
|
||||
/// <summary>
|
||||
/// The outcome of one <see cref="DeployedSecretsMirror.ReconcileAsync"/> pass: the secret
|
||||
/// names applied (inserted or updated in this node's own secret store), the names removed
|
||||
/// (deleted because central tombstoned them and this node had previously mirrored them), and
|
||||
/// a human-readable failure message per live secret that could not be applied and/or verified
|
||||
/// on this node.
|
||||
/// </summary>
|
||||
/// <param name="Applied">Names applied (inserted or updated) this pass.</param>
|
||||
/// <param name="Removed">Names deleted this pass because they were previously mirrored and are now tombstoned upstream.</param>
|
||||
/// <param name="Failed">One message per live secret that failed to apply and/or decrypt on this node — never includes secret values.</param>
|
||||
/// <param name="FailedNames">
|
||||
/// The secret name behind each <paramref name="Failed"/> entry, in the SAME order (index-for-
|
||||
/// index — each pass reports at most one failure per row, so the two lists always stay in
|
||||
/// lockstep). Added for <c>DeploymentManagerActor.HandleDeployArtifacts</c> (Task 10): this
|
||||
/// mirror reconciles the WHOLE site-wide <c>deployed_secrets</c> table every pass, not only the
|
||||
/// names a particular deploy just shipped, so a caller that needs to know whether a failure
|
||||
/// belongs to ITS OWN shipped names — as opposed to some other, previously-deployed secret, or a
|
||||
/// removal of a now-unreferenced one — cannot recover that from the message strings alone.
|
||||
/// </param>
|
||||
public sealed record MirrorResult(
|
||||
IReadOnlyList<string> Applied,
|
||||
IReadOnlyList<string> Removed,
|
||||
IReadOnlyList<string> Failed,
|
||||
IReadOnlyList<string> FailedNames)
|
||||
{
|
||||
/// <summary>Whether this pass applied or removed anything (used to decide the summary log level).</summary>
|
||||
public bool Changed => Applied.Count > 0 || Removed.Count > 0;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Per-node reconcile of the site-wide, LocalDb-replicated <c>deployed_secrets</c> table into
|
||||
/// THIS node's own <c>ZB.MOM.WW.Secrets</c> store. Design §3.5 step 2.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// Runs on EVERY site node — not a singleton — because <c>deployed_secrets</c> row content
|
||||
/// (ciphertext) is site-wide (replicated by LocalDb), but which names THIS node has actually
|
||||
/// applied into its own local secret store is node-local fact: <c>mirrored_secrets</c> is
|
||||
/// deliberately NOT one of LocalDb's replicated tables (see
|
||||
/// <see cref="SiteStorageService.GetMirroredSecretsAsync"/>). Each node reconciles for
|
||||
/// itself, and — because the two nodes' secret stores are otherwise independent — each node
|
||||
/// also independently verifies that the deployed ciphertext actually decrypts under ITS OWN
|
||||
/// master key.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Apply semantics and the confirm-then-ledger fix (fix round 1, review Important #1).</b>
|
||||
/// <see cref="ISecretStore.ApplyReplicatedAsync"/> applies last-writer-wins strictly on
|
||||
/// (<c>UpdatedUtc</c>, <c>Revision</c>) — see
|
||||
/// <see cref="ZB.MOM.WW.Secrets.Abstractions.SecretLastWriterWins"/> — and a row that is not
|
||||
/// strictly newer than the local row is silently ignored, so this mirror never trusts the call
|
||||
/// blindly: every apply is followed by an <see cref="ISecretStore.GetAsync"/> read-back, and
|
||||
/// <c>mirrored_secrets</c> is ledgered only when the local row is confirmed LIVE and its
|
||||
/// ciphertext (<c>Ciphertext</c> + <c>WrappedDek</c>) matches what was just shipped. A
|
||||
/// non-ledgered outcome is reported as a <see cref="MirrorResult.Failed"/> entry (never as the
|
||||
/// decrypt-failure message, which would misdiagnose either case below) and never overwritten a
|
||||
/// second way. Which of the two outcomes applies is decided by <b>ledger ownership</b> — whether
|
||||
/// <c>mirrored_secrets</c> already has ANY entry for the name, at any revision — not merely by
|
||||
/// whether the local row happens to be a tombstone:
|
||||
/// <list type="bullet">
|
||||
/// <item>
|
||||
/// <b>This node's own ledger already owns the name</b> (a tombstone this node wrote, or a LIVE
|
||||
/// row with different ciphertext than what was just shipped — the latter can happen under
|
||||
/// site/central clock skew, or when an earlier tombstone-recovery's bumped stamp lands ahead of
|
||||
/// a since-shipped central <c>UpdatedUtc</c> — fix round 2, review "clock skew" finding).
|
||||
/// Re-stamping is safe either way: neither <c>UpdatedUtc</c> nor <c>Revision</c> is bound into
|
||||
/// the AEAD associated data — confirmed against
|
||||
/// <c>ZB.MOM.WW.Secrets.Crypto.AesGcmEnvelopeCipher</c>, whose body layer binds only
|
||||
/// <c>SecretName.Value</c> and whose wrap layer binds only <c>KekId</c> — so the mirror
|
||||
/// re-applies the exact shipped ciphertext with <c>UpdatedUtc = max(now, local.UpdatedUtc + 1
|
||||
/// tick)</c> and <c>Revision = local.Revision + 1</c>, which beats the current local row under
|
||||
/// LWW without touching anything the cipher verifies. If that confirmed re-apply still does not
|
||||
/// read back live and matching (a concurrent conflict), the row is reported failed instead of
|
||||
/// silently left broken. A real-cipher integration test
|
||||
/// (<c>DeployedSecretsMirrorRealStackTests</c>, fix round 2) pins that a bumped-and-recovered row
|
||||
/// still decrypts to the original plaintext against the real <c>SqliteSecretStore</c> +
|
||||
/// <c>AesGcmEnvelopeCipher</c> + <c>DefaultSecretResolver</c> stack, not only by source reading.
|
||||
/// </item>
|
||||
/// <item>
|
||||
/// <b>The name is NOT in this node's ledger</b> and the local row is LIVE with different
|
||||
/// ciphertext — a genuinely out-of-band secret created directly on this node under the same
|
||||
/// name, stamped strictly newer than what central shipped. It wins LWW legitimately. This mirror
|
||||
/// does NOT overwrite it, does NOT ledger it (so a later central tombstone of the name can never
|
||||
/// delete someone's out-of-band secret via this mirror), and does NOT run the decrypt-verify
|
||||
/// against it (verifying would trivially "succeed" against the operator's own value and mask
|
||||
/// that the deployed revision never took effect).
|
||||
/// </item>
|
||||
/// </list>
|
||||
/// <b>Drift at the same revision (final review I1).</b> A name already ledgered at the shipped
|
||||
/// revision is still read back every pass: if an operator deleted it locally, or edited its value,
|
||||
/// the local row no longer matches the shipped ciphertext and is healed exactly like the owned
|
||||
/// re-stamp case above (bumped stamp, apply, read-back, cache invalidation) before the verify —
|
||||
/// so a local delete is repaired rather than misreported as a master-key mismatch, and a local
|
||||
/// edit is reverted to the deployed value.
|
||||
/// Ledgering always uses the SHIPPED artifact's own revision (not any bumped local revision) so
|
||||
/// a later pass's "is this ledgered at the shipped revision" check stays correct.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Per-row isolation (review Important #2).</b> Each row's apply/verify body runs in its own
|
||||
/// try/catch; an exception — an unrecognised <c>ContentType</c> from a newer central, a
|
||||
/// per-row store fault, a malformed name — is caught, reported as a <see cref="MirrorResult.Failed"/>
|
||||
/// entry naming the secret and the exception type/message (never a value), logged with the same
|
||||
/// de-spam rule, and does not stop the rest of the pass. Only an exception attributable to the
|
||||
/// caller's own cancellation (<c>ct.IsCancellationRequested</c>) propagates.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Cache invalidation (review Important #3).</b> The concrete resolver
|
||||
/// (<c>DefaultSecretResolver</c>) caches decrypted plaintext for a TTL and also implements
|
||||
/// <see cref="ISecretCacheInvalidator"/>; every ZB.MOM.WW.Secrets write path invalidates after a
|
||||
/// write for exactly this reason. This mirror does the same: after every successful apply
|
||||
/// (including a tombstone re-materialization) or delete, and before the verify step, it calls
|
||||
/// <see cref="ISecretCacheInvalidator.Invalidate"/> (a <c>null</c> invalidator is tolerated for
|
||||
/// direct construction; under DI <c>AddZbSecrets</c> always registers one) so neither the verify nor a script running immediately afterward
|
||||
/// can observe a stale pre-rotation plaintext for up to the cache TTL.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Node-local store assumption (review Minor #5; Task 11 removed every replication mode).</b>
|
||||
/// This mirror assumes the site's <see cref="ISecretStore"/> is always plain, node-local, local
|
||||
/// SQLite — the ONLY shape a Site node's store can be after Task 11 retired both the pull-only
|
||||
/// gRPC secrets hub and the SQL-Server <c>ReplicatingSecretStore</c> mode that preceded it (see
|
||||
/// <c>SecretsRegistration</c>). It would NOT have been safe to run this mirror against a store
|
||||
/// decorated by either retired replication mode: <c>DeleteAsync</c> would have published the
|
||||
/// tombstone to every other node sharing the decorator, so a site merely un-referencing a secret
|
||||
/// would have deleted it fleet-wide. <c>StartupValidator</c> now rejects
|
||||
/// <c>Secrets:Replication:Enabled=true</c> outright on either role, so that configuration can no
|
||||
/// longer reach this mirror at all.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Resolve semantics and the resolver cache.</b> The trial-resolve verify step goes through
|
||||
/// <see cref="ISecretResolver.GetAsync"/>, which the concrete resolver caches for a TTL
|
||||
/// (<c>ResolveCacheTtl</c>, 30 s in the shipped implementation). Because every successful apply
|
||||
/// or delete now invalidates that cache first (see above), the remaining exposure is narrow: a
|
||||
/// KEK rotation on this node's OWN master key (not a secret rotation at central) landing inside
|
||||
/// the cache window of an unrelated resolve. That residual case is still an acceptable false
|
||||
/// negative for the "did the key change" question — the cache hit still proves this node once
|
||||
/// decrypted a value with a key that produced that cache entry — and is not a bug in this
|
||||
/// mirror.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Error-log de-spam.</b> A secret that keeps failing (to apply OR to decrypt) would
|
||||
/// otherwise log an Error line every reconcile pass forever. This mirror logs Error only the
|
||||
/// FIRST time a given failure fingerprint (kind + revision + relevant detail, e.g. KekId for a
|
||||
/// decrypt failure) occurs for a name on this node; identical repeats log at Debug instead. The
|
||||
/// moment that same name applies/resolves successfully, the remembered fingerprint is cleared,
|
||||
/// so a LATER failure — even one with an identical fingerprint — logs at Error again. This state
|
||||
/// is in-memory only (per node process), which matches the ledger's own node-local scope.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class DeployedSecretsMirror : BackgroundService
|
||||
{
|
||||
private const string DeleteActor = "deployed-secrets-mirror";
|
||||
|
||||
private readonly SiteStorageService _storage;
|
||||
private readonly ISecretStore _store;
|
||||
private readonly ISecretResolver _resolver;
|
||||
private readonly ISecretCacheInvalidator? _invalidator;
|
||||
private readonly IOptions<SiteRuntimeOptions> _options;
|
||||
private readonly ILogger<DeployedSecretsMirror> _logger;
|
||||
private readonly TimeProvider _time;
|
||||
|
||||
private readonly SemaphoreSlim _gate = new(1, 1);
|
||||
private bool _disposed;
|
||||
|
||||
/// <summary>
|
||||
/// Per-secret-name record of the failure fingerprint this node has already logged at Error
|
||||
/// (an apply-conflict or a decrypt failure — see the class remarks). Cleared for a name the
|
||||
/// moment it applies/resolves successfully, so a later failure — even an identical one —
|
||||
/// logs at Error again. In-memory only; not a durable ledger, and deliberately so (it exists
|
||||
/// purely to de-spam this process's logs).
|
||||
/// </summary>
|
||||
private readonly Dictionary<string, string> _lastLoggedFailure = new(StringComparer.Ordinal);
|
||||
|
||||
/// <summary>
|
||||
/// Names whose same-revision local drift (final review I1) has already been logged at
|
||||
/// Warning and not yet healed — a heal that keeps failing logs its drift at Debug on repeats,
|
||||
/// matching the <see cref="_lastLoggedFailure"/> de-spam rule. Cleared on a successful apply.
|
||||
/// </summary>
|
||||
private readonly HashSet<string> _driftLogged = new(StringComparer.Ordinal);
|
||||
|
||||
/// <summary>Initializes a new instance of <see cref="DeployedSecretsMirror"/>.</summary>
|
||||
/// <param name="storage">Site-local storage exposing <c>deployed_secrets</c> and the <c>mirrored_secrets</c> ledger.</param>
|
||||
/// <param name="store">This node's own secret store.</param>
|
||||
/// <param name="resolver">This node's own secret resolver, used to trial-decrypt each deployed secret.</param>
|
||||
/// <param name="invalidator">
|
||||
/// Cache invalidator for <paramref name="resolver"/> (the concrete
|
||||
/// <c>DefaultSecretResolver</c> implements this itself). The parameter has no default, so
|
||||
/// under MS DI activation fails unless an <see cref="ISecretCacheInvalidator"/> is registered
|
||||
/// — <c>AddZbSecrets</c> always registers one. A caller constructing the mirror directly may
|
||||
/// pass <see langword="null"/>, in which case no invalidation is attempted.
|
||||
/// </param>
|
||||
/// <param name="options">Supplies <see cref="SiteRuntimeOptions.DeployedSecretsMirrorInterval"/>.</param>
|
||||
/// <param name="logger">Logger instance. Never logs secret values — names, revisions, and KEK ids only.</param>
|
||||
/// <param name="time">Clock driving the reconcile loop's <see cref="PeriodicTimer"/> and the tombstone-recovery stamp.</param>
|
||||
public DeployedSecretsMirror(
|
||||
SiteStorageService storage,
|
||||
ISecretStore store,
|
||||
ISecretResolver resolver,
|
||||
ISecretCacheInvalidator? invalidator,
|
||||
IOptions<SiteRuntimeOptions> options,
|
||||
ILogger<DeployedSecretsMirror> logger,
|
||||
TimeProvider time)
|
||||
{
|
||||
_storage = storage ?? throw new ArgumentNullException(nameof(storage));
|
||||
_store = store ?? throw new ArgumentNullException(nameof(store));
|
||||
_resolver = resolver ?? throw new ArgumentNullException(nameof(resolver));
|
||||
_invalidator = invalidator;
|
||||
_options = options ?? throw new ArgumentNullException(nameof(options));
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
_time = time ?? throw new ArgumentNullException(nameof(time));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Runs one reconcile pass: applies every live <c>deployed_secrets</c> row into this
|
||||
/// node's own secret store (confirmed by read-back before ledgering), deletes rows this
|
||||
/// node previously mirrored that are now tombstoned upstream, and trial-decrypts every
|
||||
/// live, confirmed-applied row to verify it resolves under this node's own master key.
|
||||
/// Safe to call concurrently with itself (and with the hosted loop) — a single in-flight
|
||||
/// pass is enforced internally.
|
||||
/// </summary>
|
||||
/// <param name="ct">A token to cancel the operation.</param>
|
||||
/// <returns>The pass's <see cref="MirrorResult"/>.</returns>
|
||||
public async Task<MirrorResult> ReconcileAsync(CancellationToken ct)
|
||||
{
|
||||
await _gate.WaitAsync(ct).ConfigureAwait(false);
|
||||
try
|
||||
{
|
||||
// Tolerant read (final review M4): a row whose row_json cannot be parsed is handed
|
||||
// back separately instead of throwing out of the whole pass, so it is reported
|
||||
// against its own name like any other per-row failure.
|
||||
var (rows, unreadable) = await _storage.GetDeployedSecretsTolerantAsync().ConfigureAwait(false);
|
||||
var mirrored = (await _storage.GetMirroredSecretsAsync().ConfigureAwait(false))
|
||||
.ToDictionary(m => m.Name, m => m.Revision, StringComparer.Ordinal);
|
||||
|
||||
List<string> applied = [];
|
||||
List<string> removed = [];
|
||||
List<string> failed = [];
|
||||
List<string> failedNames = [];
|
||||
|
||||
foreach (var row in rows)
|
||||
{
|
||||
try
|
||||
{
|
||||
if (row.IsRemoved)
|
||||
{
|
||||
await ProcessRemovedRowAsync(row.Name, mirrored, removed, ct).ConfigureAwait(false);
|
||||
}
|
||||
else
|
||||
{
|
||||
await ProcessLiveRowAsync(row, mirrored, applied, failed, failedNames, ct).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
catch (Exception ex) when (!ct.IsCancellationRequested)
|
||||
{
|
||||
ReportFailure(
|
||||
row.Name,
|
||||
$"error:{row.Secret.Revision}:{ex.GetType().Name}:{ex.Message}",
|
||||
$"Secret '{row.Name}' failed to process on this node ({ex.GetType().Name}: {ex.Message})",
|
||||
failed,
|
||||
failedNames);
|
||||
}
|
||||
}
|
||||
|
||||
foreach (var bad in unreadable)
|
||||
{
|
||||
try
|
||||
{
|
||||
if (bad.IsRemoved)
|
||||
{
|
||||
// A removal needs only the name, so an unreadable tombstone is still
|
||||
// honoured for a name this node previously mirrored.
|
||||
await ProcessRemovedRowAsync(bad.Name, mirrored, removed, ct).ConfigureAwait(false);
|
||||
}
|
||||
else
|
||||
{
|
||||
ReportFailure(
|
||||
bad.Name,
|
||||
$"unreadable:{bad.Error}",
|
||||
$"Secret '{bad.Name}' could not be read from deployed_secrets on this node ({bad.Error})",
|
||||
failed,
|
||||
failedNames);
|
||||
}
|
||||
}
|
||||
catch (Exception ex) when (!ct.IsCancellationRequested)
|
||||
{
|
||||
ReportFailure(
|
||||
bad.Name,
|
||||
$"error:unreadable:{ex.GetType().Name}:{ex.Message}",
|
||||
$"Secret '{bad.Name}' failed to process on this node ({ex.GetType().Name}: {ex.Message})",
|
||||
failed,
|
||||
failedNames);
|
||||
}
|
||||
}
|
||||
|
||||
var result = new MirrorResult(applied, removed, failed, failedNames);
|
||||
|
||||
if (result.Changed)
|
||||
{
|
||||
_logger.LogInformation(
|
||||
"Deployed secrets mirror: {Applied} applied, {Removed} removed, {Failed} failed",
|
||||
applied.Count, removed.Count, failed.Count);
|
||||
}
|
||||
else
|
||||
{
|
||||
_logger.LogDebug(
|
||||
"Deployed secrets mirror in sync ({Count} secret(s), {FailedCount} failing)",
|
||||
rows.Count(r => !r.IsRemoved) + unreadable.Count(u => !u.IsRemoved), failed.Count);
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
finally
|
||||
{
|
||||
_gate.Release();
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Handles a tombstoned <c>deployed_secrets</c> row: deletes it from this node's store, but
|
||||
/// only if this node had previously ledgered it as mirrored — an out-of-band, same-named
|
||||
/// local secret this mirror never applied is left alone.
|
||||
/// </summary>
|
||||
private async Task ProcessRemovedRowAsync(
|
||||
string rowName, Dictionary<string, long> mirrored, List<string> removed, CancellationToken ct)
|
||||
{
|
||||
if (!mirrored.ContainsKey(rowName))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
await _store.DeleteAsync(new SecretName(rowName), DeleteActor, ct).ConfigureAwait(false);
|
||||
_invalidator?.Invalidate(new SecretName(rowName));
|
||||
await _storage.RemoveMirroredSecretAsync(rowName).ConfigureAwait(false);
|
||||
removed.Add(rowName);
|
||||
_lastLoggedFailure.Remove(rowName);
|
||||
_logger.LogInformation(
|
||||
"Removed deployed secret {SecretName} (no longer referenced by this site)",
|
||||
rowName);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Handles a live <c>deployed_secrets</c> row: applies it (confirmed by read-back) when not
|
||||
/// already ledgered at the shipped revision, or when it is ledgered but the local row has
|
||||
/// since gone missing, been tombstoned or drifted to different ciphertext (healed via the
|
||||
/// mirror-owned re-stamp path), then verifies it decrypts — unless the apply
|
||||
/// outcome makes a verify meaningless (an out-of-band local secret won LWW), in which case a
|
||||
/// single, distinct failure is reported instead.
|
||||
/// </summary>
|
||||
private async Task ProcessLiveRowAsync(
|
||||
DeployedSecretRow row,
|
||||
Dictionary<string, long> mirrored,
|
||||
List<string> applied,
|
||||
List<string> failed,
|
||||
List<string> failedNames,
|
||||
CancellationToken ct)
|
||||
{
|
||||
var name = new SecretName(row.Name);
|
||||
// "Owned by this mirror" is ANY ledger entry for the name, at whatever revision — not
|
||||
// only a match on the shipped revision. Ownership, not revision equality, is what
|
||||
// decides whether a live-but-mismatched local row is OUR stale stamp (safe to bump past)
|
||||
// or someone else's out-of-band value (never touched) — see ApplyAndConfirmAsync and
|
||||
// the class remarks (fix round 2, review Minor "clock skew").
|
||||
var ownedByMirror = mirrored.TryGetValue(row.Name, out var mirroredRevision);
|
||||
var alreadyLedgered = ownedByMirror && mirroredRevision == row.Secret.Revision;
|
||||
|
||||
// Ledgered at the shipped revision is NOT proof the local row still holds it (final
|
||||
// review I1): an operator may have deleted or edited this mirror-owned name on the node
|
||||
// since. Read it back; only a live row whose ciphertext matches the shipped artifact goes
|
||||
// straight to verify. A missing, tombstoned or drifted row is healed through the same
|
||||
// mirror-owned re-stamp path as any other owned mismatch — central is authoritative for
|
||||
// every name it deploys — instead of being misreported as a master-key mismatch.
|
||||
var needsApply = !alreadyLedgered;
|
||||
if (alreadyLedgered)
|
||||
{
|
||||
var local = await _store.GetAsync(name, ct).ConfigureAwait(false);
|
||||
if (local is null || local.IsDeleted || !MatchesShippedCiphertext(local, ToStored(row.Secret)))
|
||||
{
|
||||
needsApply = true;
|
||||
_logger.Log(
|
||||
_driftLogged.Add(row.Name) ? LogLevel.Warning : LogLevel.Debug,
|
||||
"Deployed secret {SecretName} r{Revision} was {Drift} on this node; re-applying the deployed revision",
|
||||
row.Name,
|
||||
row.Secret.Revision,
|
||||
local is null ? "missing" : local.IsDeleted ? "deleted" : "changed");
|
||||
}
|
||||
}
|
||||
|
||||
if (needsApply)
|
||||
{
|
||||
var outcome = await ApplyAndConfirmAsync(row, name, ownedByMirror, ct).ConfigureAwait(false);
|
||||
|
||||
switch (outcome)
|
||||
{
|
||||
case ApplyOutcome.Applied:
|
||||
await _storage.SetMirroredSecretAsync(row.Name, row.Secret.Revision).ConfigureAwait(false);
|
||||
_invalidator?.Invalidate(name);
|
||||
applied.Add(row.Name);
|
||||
_lastLoggedFailure.Remove(row.Name);
|
||||
_driftLogged.Remove(row.Name);
|
||||
_logger.LogInformation(
|
||||
"Applied deployed secret {SecretName} r{Revision} (kek {KekId})",
|
||||
row.Name, row.Secret.Revision, row.Secret.KekId);
|
||||
break;
|
||||
|
||||
case ApplyOutcome.LocalNewer:
|
||||
ReportFailure(
|
||||
row.Name,
|
||||
$"outofband:{row.Secret.Revision}",
|
||||
$"Secret '{row.Name}' exists on this node with a newer local value; " +
|
||||
"the deployed revision was not applied",
|
||||
failed,
|
||||
failedNames);
|
||||
return; // Do not verify — that would decrypt the operator's own value.
|
||||
|
||||
case ApplyOutcome.Unconfirmed:
|
||||
ReportFailure(
|
||||
row.Name,
|
||||
$"unconfirmed:{row.Secret.Revision}",
|
||||
$"Secret '{row.Name}' could not be confirmed applied on this node " +
|
||||
"after a re-apply attempt",
|
||||
failed,
|
||||
failedNames);
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
await VerifyAsync(row, failed, failedNames, ct).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <summary>The confirmed outcome of applying one shipped row into this node's store. See the class remarks.</summary>
|
||||
private enum ApplyOutcome
|
||||
{
|
||||
/// <summary>The local row is now live and its ciphertext matches what was shipped.</summary>
|
||||
Applied,
|
||||
|
||||
/// <summary>The local row is live with DIFFERENT ciphertext — an out-of-band secret that legitimately won LWW.</summary>
|
||||
LocalNewer,
|
||||
|
||||
/// <summary>Neither of the above — a tombstone re-apply attempt did not read back as applied.</summary>
|
||||
Unconfirmed,
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Calls <see cref="ISecretStore.ApplyReplicatedAsync"/> and confirms the outcome by reading
|
||||
/// the row back — see the class remarks for why this mirror never trusts the call blindly.
|
||||
/// </summary>
|
||||
/// <param name="row">The shipped row to apply.</param>
|
||||
/// <param name="name">The row's <see cref="SecretName"/>.</param>
|
||||
/// <param name="ownedByMirror">
|
||||
/// Whether this node's ledger already has ANY entry for this name (at any revision) — i.e.
|
||||
/// this mirror, not an operator, is the one that put the current local row there. A
|
||||
/// live-but-mismatched local row is only ever re-stamped (never reported as out-of-band)
|
||||
/// when this is <see langword="true"/>.
|
||||
/// </param>
|
||||
/// <param name="ct">A token to cancel the operation.</param>
|
||||
private async Task<ApplyOutcome> ApplyAndConfirmAsync(
|
||||
DeployedSecretRow row, SecretName name, bool ownedByMirror, CancellationToken ct)
|
||||
{
|
||||
var shipped = ToStored(row.Secret);
|
||||
await _store.ApplyReplicatedAsync(shipped, ct).ConfigureAwait(false);
|
||||
|
||||
var local = await _store.GetAsync(name, ct).ConfigureAwait(false);
|
||||
|
||||
if (local is not null && !local.IsDeleted && MatchesShippedCiphertext(local, shipped))
|
||||
{
|
||||
return ApplyOutcome.Applied;
|
||||
}
|
||||
|
||||
if (local is not null && (local.IsDeleted || ownedByMirror))
|
||||
{
|
||||
// Either a tombstone (this node deleted it, or some earlier apply left it that way),
|
||||
// or a LIVE row that is nonetheless OURS — the ledger already has an entry for this
|
||||
// name, so the mismatch is this mirror's own earlier stamp (e.g. a prior tombstone
|
||||
// recovery's bumped UpdatedUtc landing ahead of a central UpdatedUtc that arrives
|
||||
// later, under site/central clock skew — review "clock skew" finding), not an
|
||||
// operator's out-of-band value. Both are safe to re-stamp: neither UpdatedUtc nor
|
||||
// Revision is bound into the AEAD associated data (confirmed against
|
||||
// AesGcmEnvelopeCipher — only SecretName and KekId are AAD), so re-stamping strictly
|
||||
// past the current local row changes nothing the cipher verifies.
|
||||
var nowUtc = _time.GetUtcNow();
|
||||
var localUtcPlusOneTick = local.UpdatedUtc.AddTicks(1);
|
||||
var bumpedUpdatedUtc = nowUtc > localUtcPlusOneTick ? nowUtc : localUtcPlusOneTick;
|
||||
var bumped = shipped with { UpdatedUtc = bumpedUpdatedUtc, Revision = local.Revision + 1 };
|
||||
|
||||
await _store.ApplyReplicatedAsync(bumped, ct).ConfigureAwait(false);
|
||||
var reconfirmed = await _store.GetAsync(name, ct).ConfigureAwait(false);
|
||||
|
||||
return reconfirmed is not null && !reconfirmed.IsDeleted && MatchesShippedCiphertext(reconfirmed, shipped)
|
||||
? ApplyOutcome.Applied
|
||||
: ApplyOutcome.Unconfirmed;
|
||||
}
|
||||
|
||||
// local is live, NOT ours (never ledgered), and its ciphertext differs from what was
|
||||
// shipped: a genuine out-of-band value with a strictly newer (UpdatedUtc, Revision) won
|
||||
// LWW legitimately. (local is null only in a pathological concurrent-delete race;
|
||||
// Unconfirmed covers that too.)
|
||||
return local is null ? ApplyOutcome.Unconfirmed : ApplyOutcome.LocalNewer;
|
||||
}
|
||||
|
||||
private static bool MatchesShippedCiphertext(StoredSecret local, StoredSecret shipped) =>
|
||||
local.Ciphertext.AsSpan().SequenceEqual(shipped.Ciphertext)
|
||||
&& local.WrappedDek.AsSpan().SequenceEqual(shipped.WrappedDek);
|
||||
|
||||
/// <summary>
|
||||
/// Trial-resolves <paramref name="row"/>'s secret through <see cref="ISecretResolver"/> to
|
||||
/// verify it decrypts on this node. On failure, appends a message to
|
||||
/// <paramref name="failed"/> and logs with the shared de-spam rule (see the class remarks).
|
||||
/// </summary>
|
||||
private async Task VerifyAsync(
|
||||
DeployedSecretRow row, List<string> failed, List<string> failedNames, CancellationToken ct)
|
||||
{
|
||||
bool resolved;
|
||||
try
|
||||
{
|
||||
resolved = await _resolver.GetAsync(new SecretName(row.Name), ct).ConfigureAwait(false) is not null;
|
||||
}
|
||||
catch (OperationCanceledException) when (ct.IsCancellationRequested)
|
||||
{
|
||||
throw;
|
||||
}
|
||||
catch (Exception)
|
||||
{
|
||||
resolved = false;
|
||||
}
|
||||
|
||||
if (resolved)
|
||||
{
|
||||
_lastLoggedFailure.Remove(row.Name);
|
||||
return;
|
||||
}
|
||||
|
||||
var message =
|
||||
$"Secret '{row.Name}' (kek {row.Secret.KekId}) did not decrypt on this node — " +
|
||||
"the node's master key differs from central's";
|
||||
ReportFailure(row.Name, $"decrypt:{row.Secret.Revision}:{row.Secret.KekId}", message, failed, failedNames);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Appends <paramref name="message"/> to <paramref name="failed"/> (and <paramref name="name"/>
|
||||
/// to <paramref name="failedNames"/>, in lockstep) and logs it — Error the first time
|
||||
/// <paramref name="fingerprint"/> is seen for <paramref name="name"/> since it last
|
||||
/// succeeded, Debug on every identical repeat.
|
||||
/// </summary>
|
||||
private void ReportFailure(
|
||||
string name, string fingerprint, string message, List<string> failed, List<string> failedNames)
|
||||
{
|
||||
failed.Add(message);
|
||||
failedNames.Add(name);
|
||||
|
||||
if (_lastLoggedFailure.TryGetValue(name, out var previous) && previous == fingerprint)
|
||||
{
|
||||
_logger.LogDebug("{Failure}", message);
|
||||
}
|
||||
else
|
||||
{
|
||||
_lastLoggedFailure[name] = fingerprint;
|
||||
_logger.LogError("{Failure}", message);
|
||||
}
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
|
||||
{
|
||||
using var timer = new PeriodicTimer(_options.Value.DeployedSecretsMirrorInterval, _time);
|
||||
do
|
||||
{
|
||||
try
|
||||
{
|
||||
await ReconcileAsync(stoppingToken).ConfigureAwait(false);
|
||||
}
|
||||
catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
break;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
// Covers every other failure, including an OperationCanceledException NOT
|
||||
// attributable to this service's own shutdown token (e.g. a store/command
|
||||
// timeout) — that must not silently end the loop (review Minor #4).
|
||||
_logger.LogWarning(ex, "Deployed secrets mirror pass failed; retrying next interval.");
|
||||
}
|
||||
} while (await timer.WaitForNextTickAsync(stoppingToken).ConfigureAwait(false));
|
||||
}
|
||||
|
||||
/// <summary>Maps a wire <see cref="SecretArtifact"/> row back to the store's <see cref="StoredSecret"/> shape.</summary>
|
||||
private static StoredSecret ToStored(SecretArtifact secret) => new()
|
||||
{
|
||||
Name = new SecretName(secret.Name),
|
||||
Description = secret.Description,
|
||||
ContentType = Enum.Parse<SecretContentType>(secret.ContentType, ignoreCase: false),
|
||||
Ciphertext = secret.Ciphertext,
|
||||
Nonce = secret.Nonce,
|
||||
Tag = secret.Tag,
|
||||
WrappedDek = secret.WrappedDek,
|
||||
WrapNonce = secret.WrapNonce,
|
||||
WrapTag = secret.WrapTag,
|
||||
KekId = secret.KekId,
|
||||
Revision = secret.Revision,
|
||||
CreatedUtc = secret.CreatedUtc,
|
||||
UpdatedUtc = secret.UpdatedUtc,
|
||||
};
|
||||
|
||||
/// <inheritdoc />
|
||||
public override void Dispose()
|
||||
{
|
||||
if (!_disposed)
|
||||
{
|
||||
_disposed = true;
|
||||
_gate.Dispose();
|
||||
}
|
||||
|
||||
base.Dispose();
|
||||
}
|
||||
}
|
||||
@@ -1,4 +1,5 @@
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.DependencyInjection.Extensions;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.LocalDb;
|
||||
@@ -9,6 +10,7 @@ using ZB.MOM.WW.ScadaBridge.SiteRuntime.Deployment;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Repositories;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Secrets;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Streaming;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime;
|
||||
@@ -123,6 +125,21 @@ public static class ServiceCollectionExtensions
|
||||
sp.GetRequiredService<IOptionsMonitor<SiteRuntimeOptions>>(),
|
||||
sp.GetRequiredService<ILogger<ScriptRunSummaryFlushService>>()));
|
||||
|
||||
// Site-secrets artifact deploy (Task 9): per-node reconcile of the site-wide,
|
||||
// LocalDb-replicated deployed_secrets table into THIS node's own ZB.MOM.WW.Secrets
|
||||
// store. Runs on every site node — not a singleton, since mirrored_secrets (which
|
||||
// names this node has already applied) is deliberately node-local, not replicated.
|
||||
// Registered as a singleton (not just a hosted service) so ReconcileAsync can also
|
||||
// be invoked synchronously — by tests, and by any future on-demand trigger (e.g.
|
||||
// right after an artifact deploy lands secrets) — without waiting for the next tick.
|
||||
// ISecretStore/ISecretResolver are registered by the Host's AddScadaBridgeSecrets
|
||||
// (SecretsNodeRole.Site) — DI registration order does not matter, only that both
|
||||
// calls happen before the provider is built, which SiteServiceRegistration.Configure
|
||||
// guarantees.
|
||||
services.TryAddSingleton(TimeProvider.System);
|
||||
services.AddSingleton<DeployedSecretsMirror>();
|
||||
services.AddHostedService(sp => sp.GetRequiredService<DeployedSecretsMirror>());
|
||||
|
||||
return services;
|
||||
}
|
||||
|
||||
|
||||
@@ -263,4 +263,15 @@ public class SiteRuntimeOptions
|
||||
/// allow-listed roots and per-read size cap. Off by default (no roots).
|
||||
/// </summary>
|
||||
public Scripts.ScriptFileAccessOptions ScriptFileAccess { get; set; } = new();
|
||||
|
||||
/// <summary>
|
||||
/// Interval at which <see cref="Secrets.DeployedSecretsMirror"/> reconciles the
|
||||
/// site-wide, LocalDb-replicated <c>deployed_secrets</c> table into THIS node's own
|
||||
/// <c>ZB.MOM.WW.Secrets</c> store — applying live rows, deleting rows this node
|
||||
/// previously mirrored that are now tombstoned, and trial-decrypting every live row to
|
||||
/// verify it resolves under this node's own master key. The reconcile also runs once
|
||||
/// immediately at hosted-service start, so this interval only governs the steady-state
|
||||
/// cadence. Default: 15 seconds.
|
||||
/// </summary>
|
||||
public TimeSpan DeployedSecretsMirrorInterval { get; set; } = TimeSpan.FromSeconds(15);
|
||||
}
|
||||
|
||||
@@ -131,5 +131,10 @@ public sealed class SiteRuntimeOptionsValidator : OptionsValidatorBase<SiteRunti
|
||||
$"ScadaBridge:SiteRuntime:ScriptFileAccess:AllowedRoots:{i} must be an absolute directory path " +
|
||||
$"(was '{root}'); a relative root would resolve against the service's working directory.");
|
||||
}
|
||||
|
||||
builder.RequireThat(options.DeployedSecretsMirrorInterval > TimeSpan.Zero,
|
||||
$"ScadaBridge:SiteRuntime:DeployedSecretsMirrorInterval must be a positive duration " +
|
||||
$"(was {options.DeployedSecretsMirrorInterval}); it is the DeployedSecretsMirror " +
|
||||
"reconcile PeriodicTimer period.");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -24,6 +24,7 @@
|
||||
<PackageReference Include="Microsoft.Extensions.Options" />
|
||||
<PackageReference Include="ZB.MOM.WW.Configuration" />
|
||||
<PackageReference Include="ZB.MOM.WW.LocalDb" />
|
||||
<PackageReference Include="ZB.MOM.WW.Secrets.Abstractions" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
|
||||
@@ -13,6 +13,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
|
||||
using ZB.MOM.WW.ScadaBridge.Communication;
|
||||
using ZB.MOM.WW.ScadaBridge.DeploymentManager;
|
||||
using ZB.MOM.WW.ScadaBridge.DeploymentManager.PendingUpdates;
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
using SitesPage = ZB.MOM.WW.ScadaBridge.CentralUI.Components.Pages.Admin.Sites;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.CentralUI.Tests.Admin;
|
||||
@@ -54,7 +55,10 @@ public class SitesPageTests : BunitContext
|
||||
Substitute.For<IAuditService>(),
|
||||
Options.Create(new DeploymentManagerOptions()),
|
||||
NullLogger<ArtifactDeploymentService>.Instance,
|
||||
new ArtifactRevisionHashService());
|
||||
new ArtifactRevisionHashService(),
|
||||
// These tests don't exercise secret references, so a store that never has
|
||||
// anything is never called — CollectAsync only queries a scanned secret: reference.
|
||||
new SiteSecretsCollector(Substitute.For<ISecretStore>()));
|
||||
Services.AddSingleton(artifactSvc);
|
||||
|
||||
Services.AddSingleton<IDialogService>(Substitute.For<IDialogService>());
|
||||
|
||||
+85
@@ -0,0 +1,85 @@
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Secrets;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Commons.Tests.Types.Secrets;
|
||||
|
||||
/// <summary>
|
||||
/// <see cref="SecretReferenceScanner"/> derives, per site, the distinct <c>secret:<name></c>
|
||||
/// references found anywhere in its data connections' primary/backup JSON (design §3.1).
|
||||
/// </summary>
|
||||
public class SecretReferenceScannerTests
|
||||
{
|
||||
private static DataConnectionArtifact Dc(string name, string? primary, string? backup = null)
|
||||
=> new(name, "MxGateway", primary, backup, 3);
|
||||
|
||||
[Fact]
|
||||
public void FindsNestedRefs_CaseInsensitive_AcrossPrimaryAndBackup()
|
||||
{
|
||||
var refs = SecretReferenceScanner.Scan([
|
||||
Dc("GW1", """{"Endpoint":"http://gw:5000","ApiKey":"SECRET: mxgw-key "}"""),
|
||||
Dc("GW2", """{"a":{"b":["x","secret:mxgw-key"]}}""", """{"ApiKey":"secret:backup-key"}"""),
|
||||
]);
|
||||
Assert.Equal(["backup-key", "mxgw-key"], refs.Select(r => r.Name));
|
||||
Assert.Equal(["GW1", "GW2"], refs.Single(r => r.Name == "mxgw-key").ReferencedBy);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData("""{"ApiKey":"${secret:node-local}"}""")]
|
||||
[InlineData("""{"Note":"see secret:x in docs"}""")]
|
||||
[InlineData("""{"ApiKey":"secret:"}""")]
|
||||
[InlineData("""{"Port":5000}""")]
|
||||
[InlineData("not json")]
|
||||
[InlineData("""{"Psk":"secret:SB-GRPC-PSK-site-a"}""")]
|
||||
[InlineData("""{"Psk":"secret:sb-grpc-psk-site-a"}""")]
|
||||
public void IgnoresNonReferences(string json)
|
||||
=> Assert.Empty(SecretReferenceScanner.Scan([Dc("C", json)]));
|
||||
|
||||
[Fact]
|
||||
public void ReferencedBy_DeduplicatesWhenOneConnectionRefsTheSameSecretMultipleTimes()
|
||||
{
|
||||
var refs = SecretReferenceScanner.Scan([
|
||||
Dc(
|
||||
"GW1",
|
||||
"""{"a":"secret:mxgw-key","b":["secret:mxgw-key","secret:mxgw-key"]}""",
|
||||
"""{"ApiKey":"secret:mxgw-key"}"""),
|
||||
]);
|
||||
Assert.Equal(["mxgw-key"], refs.Select(r => r.Name));
|
||||
Assert.Equal(["GW1"], refs.Single(r => r.Name == "mxgw-key").ReferencedBy);
|
||||
}
|
||||
|
||||
// ── Shared parser (final review M3) ──
|
||||
|
||||
[Theory]
|
||||
[InlineData("secret:mxgw-key", "mxgw-key")]
|
||||
[InlineData("SECRET:mxgw-key", "mxgw-key")]
|
||||
[InlineData(" Secret: mxgw-key ", "mxgw-key")]
|
||||
[InlineData("secret:a/b/c", "a/b/c")]
|
||||
public void TryParseSecretReference_ParsesTrimmedCaseInsensitive(string value, string expected)
|
||||
{
|
||||
Assert.True(SecretReferenceScanner.TryParseSecretReference(value, out var name));
|
||||
Assert.Equal(expected, name);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(null)]
|
||||
[InlineData("")]
|
||||
[InlineData("secret:")]
|
||||
[InlineData("secret: ")]
|
||||
[InlineData("${secret:node-local}")]
|
||||
[InlineData("literal-api-key")]
|
||||
[InlineData("see secret:x in docs")]
|
||||
public void TryParseSecretReference_RejectsNonReferences(string? value)
|
||||
{
|
||||
Assert.False(SecretReferenceScanner.TryParseSecretReference(value, out var name));
|
||||
Assert.Equal(string.Empty, name);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Scan_AndParser_AgreeOnThePaddedCaseVariantName()
|
||||
{
|
||||
const string value = " SECRET: mxgw-key ";
|
||||
var shipped = Assert.Single(SecretReferenceScanner.Scan([Dc("GW", $$"""{"ApiKey":"{{value}}"}""")])).Name;
|
||||
Assert.True(SecretReferenceScanner.TryParseSecretReference(value, out var parsed));
|
||||
Assert.Equal(shipped, parsed);
|
||||
}
|
||||
}
|
||||
+85
@@ -0,0 +1,85 @@
|
||||
using Google.Protobuf.WellKnownTypes;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
|
||||
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests.Grpc;
|
||||
|
||||
/// <summary>
|
||||
/// Round-trip tests for the additive <c>Secrets</c> / <c>SecretsApplied</c> wire contract
|
||||
/// on <see cref="DeployArtifactsCommand"/> / <see cref="ArtifactDeploymentResponse"/>
|
||||
/// (proto field 9 / field 6). Site secrets design §3.3, task-2 of
|
||||
/// docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md.
|
||||
/// </summary>
|
||||
public class SiteCommandDtoMapperSecretsTests
|
||||
{
|
||||
private static SecretArtifact Sample(string name) => new(name, "text/plain",
|
||||
[1, 2, 3], [4], [5], [6, 7], [8], [9], "kek-A", 7,
|
||||
DateTimeOffset.Parse("2026-09-01T00:00:00Z"), DateTimeOffset.Parse("2026-09-02T00:00:00Z"), "d");
|
||||
|
||||
[Fact]
|
||||
public void DeployArtifacts_Secrets_RoundTrip_Verbatim()
|
||||
{
|
||||
var cmd = new DeployArtifactsCommand("dep", null, null, null, null, null, null,
|
||||
DateTimeOffset.UtcNow, [Sample("A"), Sample("B")]);
|
||||
var back = (DeployArtifactsCommand)SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(cmd));
|
||||
Assert.Equal(2, back.Secrets!.Count);
|
||||
Assert.Equal(cmd.Secrets![0].Ciphertext, back.Secrets[0].Ciphertext);
|
||||
Assert.Equal(cmd.Secrets[0].Nonce, back.Secrets[0].Nonce);
|
||||
Assert.Equal(cmd.Secrets[0].Tag, back.Secrets[0].Tag);
|
||||
Assert.Equal(cmd.Secrets[0].WrappedDek, back.Secrets[0].WrappedDek);
|
||||
Assert.Equal(cmd.Secrets[0].WrapNonce, back.Secrets[0].WrapNonce);
|
||||
Assert.Equal(cmd.Secrets[0].WrapTag, back.Secrets[0].WrapTag);
|
||||
Assert.Equal(cmd.Secrets[0].ContentType, back.Secrets[0].ContentType);
|
||||
Assert.Equal(cmd.Secrets[0].CreatedUtc, back.Secrets[0].CreatedUtc);
|
||||
Assert.Equal(cmd.Secrets[0].UpdatedUtc, back.Secrets[0].UpdatedUtc);
|
||||
Assert.Equal(cmd.Secrets[0].Description, back.Secrets[0].Description);
|
||||
Assert.Equal("kek-A", back.Secrets[1].KekId);
|
||||
Assert.Equal(7, back.Secrets[1].Revision);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void DeployArtifacts_NullSecrets_StaysNull()
|
||||
{
|
||||
var cmd = new DeployArtifactsCommand("dep", null, null, null, null, null, null, DateTimeOffset.UtcNow);
|
||||
var dto = SiteCommandDtoMapper.ToProto(cmd);
|
||||
Assert.Null(dto.Secrets);
|
||||
Assert.Null(((DeployArtifactsCommand)SiteCommandDtoMapper.FromProto(dto)).Secrets);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void DeployArtifacts_EmptySecrets_StaysEmpty_NotNull()
|
||||
{
|
||||
var cmd = new DeployArtifactsCommand("dep", null, null, null, null, null, null,
|
||||
DateTimeOffset.UtcNow, []);
|
||||
var back = (DeployArtifactsCommand)SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(cmd));
|
||||
Assert.NotNull(back.Secrets);
|
||||
Assert.Empty(back.Secrets!);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void DeployArtifacts_CommandDtoWithoutField9_MapsToNullSecrets()
|
||||
{
|
||||
// Simulates an OLD CENTRAL that never set field 9 — the DTO here carries no
|
||||
// Secrets wrapper at all, exactly as a pre-upgrade sender's wire bytes would
|
||||
// decode. FromProto must treat that absence as null, not an empty list.
|
||||
var dto = new DeployArtifactsCommandDto
|
||||
{
|
||||
DeploymentId = "dep",
|
||||
Timestamp = Timestamp.FromDateTimeOffset(DateTimeOffset.UtcNow)
|
||||
};
|
||||
|
||||
Assert.Null(dto.Secrets);
|
||||
|
||||
var back = SiteCommandDtoMapper.FromProto(dto);
|
||||
Assert.Null(back.Secrets);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(null)]
|
||||
[InlineData(3)]
|
||||
public void ArtifactResponse_SecretsApplied_RoundTrips(int? applied)
|
||||
{
|
||||
var r = new ArtifactDeploymentResponse("dep", "site-a", true, null, DateTimeOffset.UtcNow, applied);
|
||||
Assert.Equal(applied, SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(r)).SecretsApplied);
|
||||
}
|
||||
}
|
||||
@@ -90,6 +90,8 @@ internal static class SiteCommandSamples
|
||||
Add(new NotificationListArtifact("ops", ["a@x.com"]), new NotificationListArtifact("empty", []));
|
||||
Add(DataConnection(), new DataConnectionArtifact("bare", "OpcUa", null, null, 0));
|
||||
Add(Smtp(), new SmtpConfigurationArtifact("bare", "smtp", 25, "None", "f@x.com", null, null, null));
|
||||
Add(Secret(), new SecretArtifact(
|
||||
"bare", "text/plain", [], [], [], [], [], [], "kek-bare", 0, T2, T2));
|
||||
|
||||
// ── Lifecycle replies ──
|
||||
Add(new DeploymentStatusResponse("dep-1", "Site1.Pump1", DeploymentStatus.Success, null, T1),
|
||||
@@ -233,6 +235,11 @@ internal static class SiteCommandSamples
|
||||
private static SmtpConfigurationArtifact Smtp() =>
|
||||
new("Default", "smtp.host", 587, "Basic", "from@x.com", "user", "secret", "{\"tenant\":\"t\"}");
|
||||
|
||||
private static SecretArtifact Secret() =>
|
||||
new("EsgApiKey", "application/octet-stream",
|
||||
[1, 2, 3, 4], [5, 6], [7, 8, 9], [10, 11], [12], [13, 14, 15],
|
||||
"kek-2026-09", 4, T1, T2, "MES API key");
|
||||
|
||||
private static BrowseNode Node() => new("ns=2;s=Pump1.Speed", "Speed", BrowseNodeClass.Variable, false, "Double", -1, true);
|
||||
|
||||
private static BrowseNode NodeBare() => new("ns=2;s=Root", "Root", BrowseNodeClass.Object, true);
|
||||
|
||||
+37
@@ -52,6 +52,43 @@ public class MxGatewayDataConnectionSecretsTests
|
||||
Assert.Contains("mxgateway/testconn/api-key", resolver.Requested);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData("secret:mxgw-key ")]
|
||||
[InlineData(" SECRET: mxgw-key")]
|
||||
[InlineData("Secret:mxgw-key")]
|
||||
public async Task ConnectAsync_resolves_padded_or_case_variant_ref_to_the_name_the_scanner_ships(string apiKey)
|
||||
{
|
||||
// Final review M3: the adapter and central's SecretReferenceScanner share one parser,
|
||||
// so the name looked up is exactly the name that was deployed.
|
||||
var shipped = Assert.Single(ZB.MOM.WW.ScadaBridge.Commons.Types.Secrets.SecretReferenceScanner.Scan([
|
||||
new ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts.DataConnectionArtifact(
|
||||
"GW", "MxGateway", System.Text.Json.JsonSerializer.Serialize(new { ApiKey = apiKey }), null, 3),
|
||||
])).Name;
|
||||
|
||||
var fake = new FakeMxGatewayClient();
|
||||
var resolver = new FakeSecretResolver("resolved-plaintext-key");
|
||||
var adapter = new MxGatewayDataConnection(fake, NullLogger<MxGatewayDataConnection>.Instance, resolver);
|
||||
|
||||
await adapter.ConnectAsync(Details(apiKey));
|
||||
|
||||
Assert.Equal("mxgw-key", shipped);
|
||||
Assert.Equal([shipped], resolver.Requested);
|
||||
Assert.Equal("resolved-plaintext-key", fake.ConnectedWith!.ApiKey);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ConnectAsync_fails_closed_on_a_secret_ref_with_no_name()
|
||||
{
|
||||
var fake = new FakeMxGatewayClient();
|
||||
var resolver = new FakeSecretResolver("SHOULD-NOT-BE-USED");
|
||||
var adapter = new MxGatewayDataConnection(fake, NullLogger<MxGatewayDataConnection>.Instance, resolver);
|
||||
|
||||
await Assert.ThrowsAsync<InvalidOperationException>(() => adapter.ConnectAsync(Details("secret: ")));
|
||||
|
||||
Assert.Null(fake.ConnectedWith);
|
||||
Assert.Empty(resolver.Requested);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ConnectAsync_passes_literal_apikey_unchanged_and_does_not_call_resolver()
|
||||
{
|
||||
|
||||
+629
-6
@@ -2,6 +2,7 @@ using System.Collections.Concurrent;
|
||||
using System.Text.Json;
|
||||
using Akka.Actor;
|
||||
using Akka.TestKit.Xunit2;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using NSubstitute;
|
||||
@@ -12,6 +13,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
|
||||
using ZB.MOM.WW.ScadaBridge.Communication;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems;
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests;
|
||||
|
||||
@@ -27,6 +29,7 @@ public class ArtifactDeploymentServiceTests : TestKit
|
||||
private readonly INotificationRepository _notificationRepo;
|
||||
private readonly IAuditService _audit;
|
||||
private readonly ArtifactRevisionHashService _hashService = new();
|
||||
private readonly ISecretStore _secretStore;
|
||||
|
||||
public ArtifactDeploymentServiceTests()
|
||||
{
|
||||
@@ -39,8 +42,28 @@ public class ArtifactDeploymentServiceTests : TestKit
|
||||
// constructor dependency of ArtifactDeploymentService).
|
||||
_notificationRepo = Substitute.For<INotificationRepository>();
|
||||
_audit = Substitute.For<IAuditService>();
|
||||
// Unconfigured GetAsync returns null (NSubstitute default), i.e. "not in the store" —
|
||||
// exactly the fail-closed posture a test wants unless it explicitly seeds a row.
|
||||
_secretStore = Substitute.For<ISecretStore>();
|
||||
}
|
||||
|
||||
private static StoredSecret SecretRow(string name, long revision, string? description = null) => new()
|
||||
{
|
||||
Name = new SecretName(name),
|
||||
Description = description,
|
||||
ContentType = SecretContentType.Text,
|
||||
Ciphertext = [1, 2, 3],
|
||||
Nonce = [4, 5],
|
||||
Tag = [6, 7],
|
||||
WrappedDek = [8, 9],
|
||||
WrapNonce = [10, 11],
|
||||
WrapTag = [12, 13],
|
||||
KekId = "kek-1",
|
||||
Revision = revision,
|
||||
CreatedUtc = DateTimeOffset.Parse("2026-01-01T00:00:00Z"),
|
||||
UpdatedUtc = DateTimeOffset.Parse("2026-01-02T00:00:00Z"),
|
||||
};
|
||||
|
||||
[Fact]
|
||||
public async Task DeployToAllSitesAsync_NoSites_ReturnsFailure()
|
||||
{
|
||||
@@ -447,6 +470,46 @@ public class ArtifactDeploymentServiceTests : TestKit
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ComputeCurrentArtifactHashesAsync_MatchesTheHashADeployStamps_WithSecrets()
|
||||
{
|
||||
// Final review M7: the parity above runs with no data connections. Pin it for a site
|
||||
// whose connections reference secrets, so the stamped hash (built from a command that
|
||||
// carries a non-empty Secrets list) equals the "what would it be now" hash — a
|
||||
// successful secrets deploy must read Current, not Out of date.
|
||||
var site = new Site("Site One", "site-1") { Id = 1 };
|
||||
var sites = new List<Site> { site };
|
||||
_siteRepo.GetAllSitesAsync(Arg.Any<CancellationToken>()).Returns(sites);
|
||||
_siteRepo.GetDataConnectionsBySiteIdAsync(1, Arg.Any<CancellationToken>())
|
||||
.Returns(new List<DataConnection>
|
||||
{
|
||||
new("GW1", "MxGateway", 1)
|
||||
{
|
||||
PrimaryConfiguration = """{"ApiKey":"secret:key1","Password":"secret:key2"}"""
|
||||
}
|
||||
});
|
||||
_secretStore.GetAsync(Arg.Is<SecretName>(n => n.Value == "key1"), Arg.Any<CancellationToken>())
|
||||
.Returns(Task.FromResult<StoredSecret?>(SecretRow("key1", 3)));
|
||||
_secretStore.GetAsync(Arg.Is<SecretName>(n => n.Value == "key2"), Arg.Any<CancellationToken>())
|
||||
.Returns(Task.FromResult<StoredSecret?>(SecretRow("key2", 7)));
|
||||
|
||||
var recorder = new ArtifactProbeRecorder();
|
||||
var probe = Sys.ActorOf(Props.Create(() => new SecretsAckProbeActor(
|
||||
recorder, cmd => cmd.Secrets == null ? (int?)null : cmd.Secrets.Count)));
|
||||
var service = CreateServiceWithCommActor(probe);
|
||||
|
||||
var deployResult = await service.DeployToAllSitesAsync("admin");
|
||||
Assert.True(deployResult.IsSuccess);
|
||||
Assert.Equal(1, deployResult.Value.SuccessCount);
|
||||
var shipped = Assert.Single(recorder.Received);
|
||||
Assert.Equal(2, shipped.Secrets!.Count);
|
||||
Assert.NotNull(site.ArtifactRevisionHash);
|
||||
|
||||
var current = await service.ComputeCurrentArtifactHashesAsync(sites);
|
||||
|
||||
Assert.Equal(site.ArtifactRevisionHash, current[site.Id]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ComputeCurrentArtifactHashesAsync_HoistsTheGlobalQueriesOnce()
|
||||
{
|
||||
@@ -479,7 +542,508 @@ public class ArtifactDeploymentServiceTests : TestKit
|
||||
await _templateRepo.DidNotReceive().GetAllSharedScriptsAsync(Arg.Any<CancellationToken>());
|
||||
}
|
||||
|
||||
private ArtifactDeploymentService CreateService()
|
||||
// ── Secrets ride the artifact deploy (design §3.1/§3.3) ──
|
||||
|
||||
[Fact]
|
||||
public async Task DeployToAllSites_SiteWithMissingSecret_FailsThatSiteOnly_AndSendsNothingToIt()
|
||||
{
|
||||
var sites = new List<Site>
|
||||
{
|
||||
new("Site One", "ok-site") { Id = 1 },
|
||||
new("Site Two", "bad-site") { Id = 2 }
|
||||
};
|
||||
_siteRepo.GetAllSitesAsync(Arg.Any<CancellationToken>()).Returns(sites);
|
||||
_siteRepo.GetDataConnectionsBySiteIdAsync(1, Arg.Any<CancellationToken>())
|
||||
.Returns(new List<DataConnection>());
|
||||
_siteRepo.GetDataConnectionsBySiteIdAsync(2, Arg.Any<CancellationToken>())
|
||||
.Returns(new List<DataConnection>
|
||||
{
|
||||
new("GW1", "MxGateway", 2) { PrimaryConfiguration = """{"ApiKey":"secret:missing-key"}""" }
|
||||
});
|
||||
// _secretStore is unconfigured for "missing-key" — GetAsync returns null (absent).
|
||||
|
||||
var recorder = new ArtifactProbeRecorder();
|
||||
var probe = Sys.ActorOf(Props.Create(() => new ArtifactProbeActor(recorder)));
|
||||
var service = CreateServiceWithCommActor(probe);
|
||||
|
||||
var result = await service.DeployToAllSitesAsync("admin");
|
||||
|
||||
Assert.True(result.IsSuccess);
|
||||
Assert.Equal(1, result.Value.SuccessCount);
|
||||
Assert.Equal(1, result.Value.FailureCount);
|
||||
Assert.Contains(result.Value.SiteResults, r => r.SiteId == "ok-site" && r.Success);
|
||||
var badResult = Assert.Single(result.Value.SiteResults, r => r.SiteId == "bad-site");
|
||||
Assert.False(badResult.Success);
|
||||
Assert.Equal(
|
||||
"Secret 'missing-key' referenced by data connection(s) GW1 is not in the central secret store",
|
||||
badResult.ErrorMessage);
|
||||
|
||||
// Nothing was sent to the bad site: the probe only ever saw ok-site's command.
|
||||
var sentCommand = Assert.Single(recorder.Received);
|
||||
Assert.DoesNotContain(
|
||||
sentCommand.DataConnections ?? [],
|
||||
dc => dc.Name == "GW1");
|
||||
|
||||
// The bad site keeps no stamp — it still reads "out of date" / "never deployed".
|
||||
var badSite = sites.Single(s => s.SiteIdentifier == "bad-site");
|
||||
Assert.Null(badSite.ArtifactRevisionHash);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ComputeCurrentHashes_MissingSecret_UsesMissingMarker()
|
||||
{
|
||||
var site = new Site("Site One", "site-1") { Id = 1 };
|
||||
_siteRepo.GetDataConnectionsBySiteIdAsync(1, Arg.Any<CancellationToken>())
|
||||
.Returns(new List<DataConnection>
|
||||
{
|
||||
new("GW1", "MxGateway", 1) { PrimaryConfiguration = """{"ApiKey":"secret:the-key"}""" }
|
||||
});
|
||||
var service = CreateService();
|
||||
|
||||
// The reference does not resolve yet: ComputeCurrentArtifactHashesAsync must not
|
||||
// fail the call — it folds a Missing marker into the hash instead.
|
||||
var beforeCreated = await service.ComputeCurrentArtifactHashesAsync([site]);
|
||||
|
||||
// The secret now exists in the store.
|
||||
_secretStore.GetAsync(Arg.Is<SecretName>(n => n.Value == "the-key"), Arg.Any<CancellationToken>())
|
||||
.Returns(Task.FromResult<StoredSecret?>(SecretRow("the-key", 1)));
|
||||
var afterCreated = await service.ComputeCurrentArtifactHashesAsync([site]);
|
||||
|
||||
Assert.NotEqual(beforeCreated[1], afterCreated[1]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task DeployLogs_NeverContainSecretMaterial()
|
||||
{
|
||||
var ciphertext = new byte[] { 0xDE, 0xAD, 0xBE, 0xEF, 0x13, 0x37 };
|
||||
var resolvedRow = SecretRow("present-key", 5) with { Ciphertext = ciphertext };
|
||||
var ciphertextBase64 = Convert.ToBase64String(ciphertext);
|
||||
var ciphertextHex = Convert.ToHexString(ciphertext);
|
||||
|
||||
var sites = new List<Site>
|
||||
{
|
||||
new("Site One", "ok-site") { Id = 1 },
|
||||
new("Site Two", "bad-site") { Id = 2 }
|
||||
};
|
||||
_siteRepo.GetAllSitesAsync(Arg.Any<CancellationToken>()).Returns(sites);
|
||||
_siteRepo.GetDataConnectionsBySiteIdAsync(1, Arg.Any<CancellationToken>())
|
||||
.Returns(new List<DataConnection>
|
||||
{
|
||||
new("GW1", "MxGateway", 1) { PrimaryConfiguration = """{"ApiKey":"secret:present-key"}""" }
|
||||
});
|
||||
_siteRepo.GetDataConnectionsBySiteIdAsync(2, Arg.Any<CancellationToken>())
|
||||
.Returns(new List<DataConnection>
|
||||
{
|
||||
new("GW2", "MxGateway", 2) { PrimaryConfiguration = """{"ApiKey":"secret:missing-key"}""" }
|
||||
});
|
||||
_secretStore.GetAsync(Arg.Is<SecretName>(n => n.Value == "present-key"), Arg.Any<CancellationToken>())
|
||||
.Returns(Task.FromResult<StoredSecret?>(resolvedRow));
|
||||
|
||||
var recorder = new ArtifactProbeRecorder();
|
||||
// Task 7: ok-site's command ships one secret, so the probe must ack it (SecretsApplied
|
||||
// = shipped count) for the site to reconcile as a success — an unset SecretsApplied
|
||||
// now reads as an old site (see the dedicated reconciliation tests below).
|
||||
var probe = Sys.ActorOf(Props.Create(() =>
|
||||
new SecretsAckProbeActor(recorder, cmd => cmd.Secrets == null ? (int?)null : cmd.Secrets.Count)));
|
||||
var capturingLogger = new CapturingLogger<ArtifactDeploymentService>();
|
||||
var service = CreateServiceWithCommActor(probe, capturingLogger);
|
||||
|
||||
var result = await service.DeployToAllSitesAsync("admin");
|
||||
|
||||
Assert.True(result.IsSuccess);
|
||||
Assert.Equal(1, result.Value.SuccessCount);
|
||||
Assert.Equal(1, result.Value.FailureCount);
|
||||
|
||||
// The Information/Error lines still name what shipped and what didn't ...
|
||||
Assert.Contains(capturingLogger.Messages, m => m.Contains("present-key@r5"));
|
||||
Assert.Contains(capturingLogger.Messages, m => m.Contains("missing-key"));
|
||||
Assert.Contains(capturingLogger.Messages, m => m.Contains("not in the central secret store"));
|
||||
|
||||
// ... but never the ciphertext, in any rendering, in any log message or exception text.
|
||||
Assert.DoesNotContain(capturingLogger.Messages, m => m.Contains(ciphertextBase64));
|
||||
Assert.DoesNotContain(capturingLogger.Messages, m => m.Contains(ciphertextHex));
|
||||
Assert.DoesNotContain(capturingLogger.Messages, m => m.Contains(ciphertextHex, StringComparison.OrdinalIgnoreCase));
|
||||
}
|
||||
|
||||
// ── Task 7: stamp gating on secret application + DeploySiteSecrets audit row ──
|
||||
|
||||
[Fact]
|
||||
public async Task OldSite_NullSecretsApplied_WithSecrets_FailsAndDoesNotStamp()
|
||||
{
|
||||
// The site's build predates the secrets feature: it acks the artifact deploy as a
|
||||
// whole (Success=true) but never populates SecretsApplied, silently ignoring the new
|
||||
// proto field. That must not read as "the secret was applied".
|
||||
var site = new Site("Site One", "site-1") { Id = 1 };
|
||||
_siteRepo.GetAllSitesAsync(Arg.Any<CancellationToken>()).Returns(new List<Site> { site });
|
||||
_siteRepo.GetDataConnectionsBySiteIdAsync(1, Arg.Any<CancellationToken>())
|
||||
.Returns(new List<DataConnection>
|
||||
{
|
||||
new("GW1", "MxGateway", 1) { PrimaryConfiguration = """{"ApiKey":"secret:the-key"}""" }
|
||||
});
|
||||
_secretStore.GetAsync(Arg.Is<SecretName>(n => n.Value == "the-key"), Arg.Any<CancellationToken>())
|
||||
.Returns(Task.FromResult<StoredSecret?>(SecretRow("the-key", 1)));
|
||||
|
||||
var recorder = new ArtifactProbeRecorder();
|
||||
var probe = Sys.ActorOf(Props.Create(() => new SecretsAckProbeActor(recorder, _ => null)));
|
||||
var service = CreateServiceWithCommActor(probe);
|
||||
|
||||
var result = await service.DeployToAllSitesAsync("admin");
|
||||
|
||||
Assert.True(result.IsSuccess);
|
||||
Assert.Equal(0, result.Value.SuccessCount);
|
||||
Assert.Equal(1, result.Value.FailureCount);
|
||||
var siteResult = Assert.Single(result.Value.SiteResults);
|
||||
Assert.False(siteResult.Success);
|
||||
Assert.Equal(
|
||||
"Site build predates secret deployment — upgrade the site before deploying secrets",
|
||||
siteResult.ErrorMessage);
|
||||
Assert.Null(siteResult.SecretsDeployed);
|
||||
Assert.Null(site.ArtifactRevisionHash);
|
||||
Assert.Null(site.ArtifactsDeployedAt);
|
||||
await _siteRepo.DidNotReceive().UpdateSiteAsync(site, Arg.Any<CancellationToken>());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task PartialApply_FailsAndDoesNotStamp()
|
||||
{
|
||||
// The site applied 1 of the 2 shipped secrets (a partial local failure). Success=true
|
||||
// on the wire response is not enough — the count must match exactly.
|
||||
var site = new Site("Site One", "site-1") { Id = 1 };
|
||||
_siteRepo.GetAllSitesAsync(Arg.Any<CancellationToken>()).Returns(new List<Site> { site });
|
||||
_siteRepo.GetDataConnectionsBySiteIdAsync(1, Arg.Any<CancellationToken>())
|
||||
.Returns(new List<DataConnection>
|
||||
{
|
||||
new("GW1", "MxGateway", 1)
|
||||
{
|
||||
PrimaryConfiguration = """{"ApiKey":"secret:key1","Password":"secret:key2"}"""
|
||||
}
|
||||
});
|
||||
_secretStore.GetAsync(Arg.Is<SecretName>(n => n.Value == "key1"), Arg.Any<CancellationToken>())
|
||||
.Returns(Task.FromResult<StoredSecret?>(SecretRow("key1", 1)));
|
||||
_secretStore.GetAsync(Arg.Is<SecretName>(n => n.Value == "key2"), Arg.Any<CancellationToken>())
|
||||
.Returns(Task.FromResult<StoredSecret?>(SecretRow("key2", 1)));
|
||||
|
||||
var recorder = new ArtifactProbeRecorder();
|
||||
var probe = Sys.ActorOf(Props.Create(() => new SecretsAckProbeActor(recorder, _ => 1)));
|
||||
var service = CreateServiceWithCommActor(probe);
|
||||
|
||||
var result = await service.DeployToAllSitesAsync("admin");
|
||||
|
||||
Assert.True(result.IsSuccess);
|
||||
Assert.Equal(0, result.Value.SuccessCount);
|
||||
Assert.Equal(1, result.Value.FailureCount);
|
||||
var siteResult = Assert.Single(result.Value.SiteResults);
|
||||
Assert.False(siteResult.Success);
|
||||
Assert.Equal("Site applied 1 of 2 secrets", siteResult.ErrorMessage);
|
||||
Assert.Equal(1, siteResult.SecretsDeployed);
|
||||
Assert.Null(site.ArtifactRevisionHash);
|
||||
await _siteRepo.DidNotReceive().UpdateSiteAsync(site, Arg.Any<CancellationToken>());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task OldSite_NoSecrets_StillStamps()
|
||||
{
|
||||
// No data connection references a secret at all, so the command ships zero secrets.
|
||||
// An old site's SecretsApplied=null reply must NOT gate the stamp in this case —
|
||||
// unchanged behaviour.
|
||||
var site = new Site("Site One", "site-1") { Id = 1 };
|
||||
_siteRepo.GetAllSitesAsync(Arg.Any<CancellationToken>()).Returns(new List<Site> { site });
|
||||
_siteRepo.GetDataConnectionsBySiteIdAsync(1, Arg.Any<CancellationToken>())
|
||||
.Returns(new List<DataConnection>());
|
||||
|
||||
var recorder = new ArtifactProbeRecorder();
|
||||
var probe = Sys.ActorOf(Props.Create(() => new ArtifactProbeActor(recorder)));
|
||||
var service = CreateServiceWithCommActor(probe);
|
||||
|
||||
var result = await service.DeployToAllSitesAsync("admin");
|
||||
|
||||
Assert.True(result.IsSuccess);
|
||||
Assert.Equal(1, result.Value.SuccessCount);
|
||||
var siteResult = Assert.Single(result.Value.SiteResults);
|
||||
Assert.True(siteResult.Success);
|
||||
Assert.Null(siteResult.SecretsDeployed);
|
||||
Assert.NotNull(site.ArtifactRevisionHash);
|
||||
await _siteRepo.Received(1).UpdateSiteAsync(site, Arg.Any<CancellationToken>());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task NoSecretReferences_ShipsEmptySecretsList_SoTheSiteRemovesPreviouslyDeployedOnes()
|
||||
{
|
||||
// The site applies DeployArtifactsCommand.Secrets as the FULL set: rows absent from it
|
||||
// are marked removed and the mirror deletes them, while a NULL Secrets means "central
|
||||
// shipped no secrets" and touches nothing. Dropping a site's LAST secret reference must
|
||||
// therefore ship an empty list, not null — otherwise the previously deployed secret stays
|
||||
// live on both site nodes forever (Task 13 live gate, check 5).
|
||||
var site = new Site("Site One", "site-1") { Id = 1 };
|
||||
_siteRepo.GetAllSitesAsync(Arg.Any<CancellationToken>()).Returns(new List<Site> { site });
|
||||
_siteRepo.GetDataConnectionsBySiteIdAsync(1, Arg.Any<CancellationToken>())
|
||||
.Returns(new List<DataConnection>
|
||||
{
|
||||
new("GW1", "MxGateway", 1) { PrimaryConfiguration = """{"ApiKey":"a-literal-key"}""" }
|
||||
});
|
||||
|
||||
var recorder = new ArtifactProbeRecorder();
|
||||
var probe = Sys.ActorOf(Props.Create(() => new SecretsAckProbeActor(recorder, _ => 0)));
|
||||
var service = CreateServiceWithCommActor(probe);
|
||||
|
||||
var result = await service.DeployToAllSitesAsync("admin");
|
||||
|
||||
Assert.True(result.IsSuccess);
|
||||
var cmd = Assert.Single(recorder.Received);
|
||||
Assert.NotNull(cmd.Secrets);
|
||||
Assert.Empty(cmd.Secrets!);
|
||||
Assert.Null(cmd.MissingSecrets);
|
||||
var siteResult = Assert.Single(result.Value.SiteResults);
|
||||
Assert.True(siteResult.Success);
|
||||
// Nothing shipped => nothing to confirm; SecretsDeployed stays null, and no
|
||||
// DeploySiteSecrets audit row is written for a deploy that carried no secrets.
|
||||
Assert.Null(siteResult.SecretsDeployed);
|
||||
Assert.NotNull(site.ArtifactRevisionHash);
|
||||
await _audit.DidNotReceive().LogAsync(
|
||||
Arg.Any<string>(), "DeploySiteSecrets", Arg.Any<string>(), Arg.Any<string>(),
|
||||
Arg.Any<string>(), Arg.Any<object?>(), Arg.Any<CancellationToken>());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SecretDeploy_WritesOneAuditRowPerSite_WithoutMaterial()
|
||||
{
|
||||
var ciphertext = new byte[] { 0xDE, 0xAD, 0xBE, 0xEF, 0x13, 0x37 };
|
||||
var resolvedRow = SecretRow("present-key", 3) with { Ciphertext = ciphertext };
|
||||
var ciphertextBase64 = Convert.ToBase64String(ciphertext);
|
||||
var ciphertextHex = Convert.ToHexString(ciphertext);
|
||||
|
||||
var secretSite = new Site("Secret Site", "sec-site") { Id = 1 };
|
||||
var plainSite = new Site("Plain Site", "plain-site") { Id = 2 };
|
||||
_siteRepo.GetAllSitesAsync(Arg.Any<CancellationToken>()).Returns(new List<Site> { secretSite, plainSite });
|
||||
_siteRepo.GetDataConnectionsBySiteIdAsync(1, Arg.Any<CancellationToken>())
|
||||
.Returns(new List<DataConnection>
|
||||
{
|
||||
new("GW1", "MxGateway", 1) { PrimaryConfiguration = """{"ApiKey":"secret:present-key"}""" }
|
||||
});
|
||||
_siteRepo.GetDataConnectionsBySiteIdAsync(2, Arg.Any<CancellationToken>())
|
||||
.Returns(new List<DataConnection>());
|
||||
_secretStore.GetAsync(Arg.Is<SecretName>(n => n.Value == "present-key"), Arg.Any<CancellationToken>())
|
||||
.Returns(Task.FromResult<StoredSecret?>(resolvedRow));
|
||||
|
||||
var recorder = new ArtifactProbeRecorder();
|
||||
// Correctly acks whatever it received, so this test exercises only the audit-row
|
||||
// requirement, not the reconciliation-failure paths covered above.
|
||||
var probe = Sys.ActorOf(Props.Create(() =>
|
||||
new SecretsAckProbeActor(recorder, cmd => cmd.Secrets == null ? (int?)null : cmd.Secrets.Count)));
|
||||
var service = CreateServiceWithCommActor(probe);
|
||||
|
||||
object? captured = null;
|
||||
var callCount = 0;
|
||||
_audit.When(x => x.LogAsync(
|
||||
Arg.Any<string>(), "DeploySiteSecrets", Arg.Any<string>(), Arg.Any<string>(), Arg.Any<string>(),
|
||||
Arg.Any<object>(), Arg.Any<CancellationToken>()))
|
||||
.Do(ci =>
|
||||
{
|
||||
callCount++;
|
||||
if ((string)ci[3] == "sec-site")
|
||||
{
|
||||
Assert.Equal("admin", ci[0]);
|
||||
Assert.Equal("SiteSecrets", ci[2]);
|
||||
Assert.Equal("Secret Site", ci[4]);
|
||||
captured = ci[5];
|
||||
}
|
||||
else
|
||||
{
|
||||
Assert.Fail($"Unexpected DeploySiteSecrets row for site '{ci[3]}' — only sec-site has secret refs.");
|
||||
}
|
||||
});
|
||||
|
||||
var result = await service.DeployToAllSitesAsync("admin");
|
||||
|
||||
Assert.True(result.IsSuccess);
|
||||
Assert.Equal(2, result.Value.SuccessCount);
|
||||
|
||||
// Exactly one DeploySiteSecrets row — for sec-site only; plain-site has no secret
|
||||
// references at all and gets none.
|
||||
Assert.Equal(1, callCount);
|
||||
Assert.NotNull(captured);
|
||||
var json = JsonSerializer.Serialize(captured);
|
||||
Assert.Contains("present-key", json);
|
||||
Assert.Contains("\"revision\":3", json);
|
||||
Assert.Contains("\"success\":true", json);
|
||||
|
||||
// ... but never any secret material, in any rendering.
|
||||
Assert.DoesNotContain(ciphertextBase64, json);
|
||||
Assert.DoesNotContain(ciphertextHex, json);
|
||||
Assert.DoesNotContain(ciphertextHex, json, StringComparison.OrdinalIgnoreCase);
|
||||
Assert.DoesNotContain("Ciphertext", json, StringComparison.OrdinalIgnoreCase);
|
||||
Assert.DoesNotContain("Nonce", json, StringComparison.OrdinalIgnoreCase);
|
||||
Assert.DoesNotContain("WrappedDek", json, StringComparison.OrdinalIgnoreCase);
|
||||
Assert.DoesNotContain("WrapNonce", json, StringComparison.OrdinalIgnoreCase);
|
||||
Assert.DoesNotContain("WrapTag", json, StringComparison.OrdinalIgnoreCase);
|
||||
Assert.DoesNotContain("\"tag\"", json, StringComparison.OrdinalIgnoreCase);
|
||||
}
|
||||
|
||||
// ── Task 7 fix round 1: the same gate + audit row on the RetryForSiteAsync path ──
|
||||
// (review gap: only the DeployToAllSitesAsync/sweep path had coverage above)
|
||||
|
||||
[Fact]
|
||||
public async Task RetryForSiteAsync_OldSite_NullSecretsApplied_WithSecrets_FailsAndDoesNotStamp()
|
||||
{
|
||||
var site = new Site("Retry Site", "retry-site") { Id = 7 };
|
||||
_siteRepo.GetSiteByIdAsync(7, Arg.Any<CancellationToken>()).Returns(site);
|
||||
_siteRepo.GetDataConnectionsBySiteIdAsync(7, Arg.Any<CancellationToken>())
|
||||
.Returns(new List<DataConnection>
|
||||
{
|
||||
new("GW1", "MxGateway", 7) { PrimaryConfiguration = """{"ApiKey":"secret:the-key"}""" }
|
||||
});
|
||||
_secretStore.GetAsync(Arg.Is<SecretName>(n => n.Value == "the-key"), Arg.Any<CancellationToken>())
|
||||
.Returns(Task.FromResult<StoredSecret?>(SecretRow("the-key", 1)));
|
||||
|
||||
var recorder = new ArtifactProbeRecorder();
|
||||
var probe = Sys.ActorOf(Props.Create(() => new SecretsAckProbeActor(recorder, _ => null)));
|
||||
var service = CreateServiceWithCommActor(probe);
|
||||
|
||||
var result = await service.RetryForSiteAsync(7, "retry-site", "admin");
|
||||
|
||||
Assert.True(result.IsFailure);
|
||||
Assert.Equal(
|
||||
"Site build predates secret deployment — upgrade the site before deploying secrets",
|
||||
result.Error);
|
||||
Assert.Null(site.ArtifactRevisionHash);
|
||||
Assert.Null(site.ArtifactsDeployedAt);
|
||||
await _siteRepo.DidNotReceive().UpdateSiteAsync(Arg.Any<Site>(), Arg.Any<CancellationToken>());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task RetryForSiteAsync_PartialApply_FailsAndDoesNotStamp()
|
||||
{
|
||||
var site = new Site("Retry Site", "retry-site") { Id = 7 };
|
||||
_siteRepo.GetSiteByIdAsync(7, Arg.Any<CancellationToken>()).Returns(site);
|
||||
_siteRepo.GetDataConnectionsBySiteIdAsync(7, Arg.Any<CancellationToken>())
|
||||
.Returns(new List<DataConnection>
|
||||
{
|
||||
new("GW1", "MxGateway", 7)
|
||||
{
|
||||
PrimaryConfiguration = """{"ApiKey":"secret:key1","Password":"secret:key2"}"""
|
||||
}
|
||||
});
|
||||
_secretStore.GetAsync(Arg.Is<SecretName>(n => n.Value == "key1"), Arg.Any<CancellationToken>())
|
||||
.Returns(Task.FromResult<StoredSecret?>(SecretRow("key1", 1)));
|
||||
_secretStore.GetAsync(Arg.Is<SecretName>(n => n.Value == "key2"), Arg.Any<CancellationToken>())
|
||||
.Returns(Task.FromResult<StoredSecret?>(SecretRow("key2", 1)));
|
||||
|
||||
var recorder = new ArtifactProbeRecorder();
|
||||
var probe = Sys.ActorOf(Props.Create(() => new SecretsAckProbeActor(recorder, _ => 1)));
|
||||
var service = CreateServiceWithCommActor(probe);
|
||||
|
||||
var result = await service.RetryForSiteAsync(7, "retry-site", "admin");
|
||||
|
||||
Assert.True(result.IsFailure);
|
||||
Assert.Equal("Site applied 1 of 2 secrets", result.Error);
|
||||
Assert.Null(site.ArtifactRevisionHash);
|
||||
await _siteRepo.DidNotReceive().UpdateSiteAsync(Arg.Any<Site>(), Arg.Any<CancellationToken>());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task RetryForSiteAsync_MissingSecret_FailsClosed_SendsNothing_WritesOneAuditRow()
|
||||
{
|
||||
_siteRepo.GetDataConnectionsBySiteIdAsync(7, Arg.Any<CancellationToken>())
|
||||
.Returns(new List<DataConnection>
|
||||
{
|
||||
new("GW1", "MxGateway", 7) { PrimaryConfiguration = """{"ApiKey":"secret:missing-key"}""" }
|
||||
});
|
||||
// _secretStore is unconfigured for "missing-key" — GetAsync returns null (absent).
|
||||
|
||||
var recorder = new ArtifactProbeRecorder();
|
||||
var probe = Sys.ActorOf(Props.Create(() => new ArtifactProbeActor(recorder)));
|
||||
var service = CreateServiceWithCommActor(probe);
|
||||
|
||||
object? captured = null;
|
||||
var callCount = 0;
|
||||
_audit.When(x => x.LogAsync(
|
||||
Arg.Any<string>(), "DeploySiteSecrets", Arg.Any<string>(), Arg.Any<string>(), Arg.Any<string>(),
|
||||
Arg.Any<object>(), Arg.Any<CancellationToken>()))
|
||||
.Do(ci =>
|
||||
{
|
||||
callCount++;
|
||||
Assert.Equal("admin", ci[0]);
|
||||
Assert.Equal("SiteSecrets", ci[2]);
|
||||
Assert.Equal("retry-site", ci[3]);
|
||||
Assert.Equal("retry-site", ci[4]);
|
||||
captured = ci[5];
|
||||
});
|
||||
|
||||
var result = await service.RetryForSiteAsync(7, "retry-site", "admin");
|
||||
|
||||
Assert.True(result.IsFailure);
|
||||
Assert.Equal(
|
||||
"Secret 'missing-key' referenced by data connection(s) GW1 is not in the central secret store",
|
||||
result.Error);
|
||||
|
||||
// Nothing was sent to the site at all.
|
||||
Assert.Empty(recorder.Received);
|
||||
|
||||
// Exactly one DeploySiteSecrets row, naming the missing secret, success=false.
|
||||
Assert.Equal(1, callCount);
|
||||
Assert.NotNull(captured);
|
||||
var json = JsonSerializer.Serialize(captured);
|
||||
Assert.Contains("missing-key", json);
|
||||
Assert.Contains("\"success\":false", json);
|
||||
Assert.Contains("\"secrets\":[]", json);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task RetryForSiteAsync_HappyPath_WithSecrets_StampsAndWritesOneAuditRow()
|
||||
{
|
||||
var ciphertext = new byte[] { 0xDE, 0xAD, 0xBE, 0xEF };
|
||||
var resolvedRow = SecretRow("present-key", 5) with { Ciphertext = ciphertext };
|
||||
var ciphertextBase64 = Convert.ToBase64String(ciphertext);
|
||||
|
||||
var site = new Site("Retry Site", "retry-site") { Id = 7 };
|
||||
_siteRepo.GetSiteByIdAsync(7, Arg.Any<CancellationToken>()).Returns(site);
|
||||
_siteRepo.GetDataConnectionsBySiteIdAsync(7, Arg.Any<CancellationToken>())
|
||||
.Returns(new List<DataConnection>
|
||||
{
|
||||
new("GW1", "MxGateway", 7) { PrimaryConfiguration = """{"ApiKey":"secret:present-key"}""" }
|
||||
});
|
||||
_secretStore.GetAsync(Arg.Is<SecretName>(n => n.Value == "present-key"), Arg.Any<CancellationToken>())
|
||||
.Returns(Task.FromResult<StoredSecret?>(resolvedRow));
|
||||
|
||||
var recorder = new ArtifactProbeRecorder();
|
||||
var probe = Sys.ActorOf(Props.Create(() =>
|
||||
new SecretsAckProbeActor(recorder, cmd => cmd.Secrets == null ? (int?)null : cmd.Secrets.Count)));
|
||||
var service = CreateServiceWithCommActor(probe);
|
||||
|
||||
object? captured = null;
|
||||
var callCount = 0;
|
||||
_audit.When(x => x.LogAsync(
|
||||
Arg.Any<string>(), "DeploySiteSecrets", Arg.Any<string>(), Arg.Any<string>(), Arg.Any<string>(),
|
||||
Arg.Any<object>(), Arg.Any<CancellationToken>()))
|
||||
.Do(ci =>
|
||||
{
|
||||
callCount++;
|
||||
Assert.Equal("retry-site", ci[3]);
|
||||
captured = ci[5];
|
||||
});
|
||||
|
||||
var result = await service.RetryForSiteAsync(7, "retry-site", "admin");
|
||||
|
||||
Assert.True(result.IsSuccess);
|
||||
Assert.Equal(1, result.Value.SecretsDeployed);
|
||||
Assert.Equal(
|
||||
_hashService.ComputeHash(Assert.Single(recorder.Received)),
|
||||
site.ArtifactRevisionHash);
|
||||
Assert.NotNull(site.ArtifactsDeployedAt);
|
||||
await _siteRepo.Received(1).UpdateSiteAsync(site, Arg.Any<CancellationToken>());
|
||||
|
||||
Assert.Equal(1, callCount);
|
||||
Assert.NotNull(captured);
|
||||
var json = JsonSerializer.Serialize(captured);
|
||||
Assert.Contains("present-key", json);
|
||||
Assert.Contains("\"revision\":5", json);
|
||||
Assert.Contains("\"success\":true", json);
|
||||
Assert.Contains("\"missing\":[]", json);
|
||||
Assert.DoesNotContain(ciphertextBase64, json);
|
||||
Assert.DoesNotContain("Ciphertext", json, StringComparison.OrdinalIgnoreCase);
|
||||
}
|
||||
|
||||
private ArtifactDeploymentService CreateService(ILogger<ArtifactDeploymentService>? logger = null)
|
||||
{
|
||||
var comms = new CommunicationService(
|
||||
Options.Create(new CommunicationOptions()),
|
||||
@@ -489,11 +1053,13 @@ public class ArtifactDeploymentServiceTests : TestKit
|
||||
_siteRepo, _deploymentRepo, _templateRepo, _externalSystemRepo, _notificationRepo,
|
||||
comms, _audit,
|
||||
Options.Create(new DeploymentManagerOptions()),
|
||||
NullLogger<ArtifactDeploymentService>.Instance,
|
||||
_hashService);
|
||||
logger ?? NullLogger<ArtifactDeploymentService>.Instance,
|
||||
_hashService,
|
||||
new SiteSecretsCollector(_secretStore));
|
||||
}
|
||||
|
||||
private ArtifactDeploymentService CreateServiceWithCommActor(IActorRef commActor)
|
||||
private ArtifactDeploymentService CreateServiceWithCommActor(
|
||||
IActorRef commActor, ILogger<ArtifactDeploymentService>? logger = null)
|
||||
{
|
||||
var comms = new CommunicationService(
|
||||
Options.Create(new CommunicationOptions
|
||||
@@ -510,8 +1076,40 @@ public class ArtifactDeploymentServiceTests : TestKit
|
||||
{
|
||||
ArtifactDeploymentTimeoutPerSite = TimeSpan.FromSeconds(5)
|
||||
}),
|
||||
NullLogger<ArtifactDeploymentService>.Instance,
|
||||
_hashService);
|
||||
logger ?? NullLogger<ArtifactDeploymentService>.Instance,
|
||||
_hashService,
|
||||
new SiteSecretsCollector(_secretStore));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Captures every formatted log message (and, for entries logged with an exception, the
|
||||
/// exception's <see cref="Exception.ToString"/>) so a test can assert on log CONTENT —
|
||||
/// specifically, that secret material never appears in it.
|
||||
/// </summary>
|
||||
private sealed class CapturingLogger<T> : ILogger<T>
|
||||
{
|
||||
public List<string> Messages { get; } = [];
|
||||
|
||||
public IDisposable BeginScope<TState>(TState state) where TState : notnull => NullScope.Instance;
|
||||
|
||||
public bool IsEnabled(LogLevel logLevel) => true;
|
||||
|
||||
public void Log<TState>(
|
||||
LogLevel logLevel, EventId eventId, TState state, Exception? exception,
|
||||
Func<TState, Exception?, string> formatter)
|
||||
{
|
||||
Messages.Add(formatter(state, exception));
|
||||
if (exception is not null)
|
||||
{
|
||||
Messages.Add(exception.ToString());
|
||||
}
|
||||
}
|
||||
|
||||
private sealed class NullScope : IDisposable
|
||||
{
|
||||
public static readonly NullScope Instance = new();
|
||||
public void Dispose() { }
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -550,4 +1148,29 @@ public class ArtifactDeploymentServiceTests : TestKit
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Task 7: stand-in CentralCommunicationActor that always reports the deploy itself as
|
||||
/// successful but lets a test control exactly what <see cref="ArtifactDeploymentResponse.SecretsApplied"/>
|
||||
/// comes back for a given command — an old site (<c>null</c>), a partial apply (a count
|
||||
/// short of what was shipped), or a correct ack (the shipped count) — so the secrets-
|
||||
/// reconciliation gate can be exercised independently of the wire-level success/failure
|
||||
/// the shared <see cref="ArtifactProbeActor"/> models.
|
||||
/// </summary>
|
||||
private class SecretsAckProbeActor : ReceiveActor
|
||||
{
|
||||
public SecretsAckProbeActor(ArtifactProbeRecorder recorder, Func<DeployArtifactsCommand, int?> secretsApplied)
|
||||
{
|
||||
Receive<SiteEnvelope>(env =>
|
||||
{
|
||||
if (env.Message is DeployArtifactsCommand cmd)
|
||||
{
|
||||
recorder.Received.Add(cmd);
|
||||
Sender.Tell(new ArtifactDeploymentResponse(
|
||||
cmd.DeploymentId, env.SiteId, true, null, DateTimeOffset.UtcNow,
|
||||
secretsApplied(cmd)));
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+59
@@ -46,6 +46,65 @@ public class ArtifactRevisionHashServiceTests
|
||||
SmtpConfigurations: null,
|
||||
timestamp ?? DateTimeOffset.UtcNow);
|
||||
|
||||
private static SecretArtifact Secret(string name, long rev) =>
|
||||
new(name, "text/plain",
|
||||
Ciphertext: [1, 2, 3],
|
||||
Nonce: [4, 5],
|
||||
Tag: [6, 7],
|
||||
WrappedDek: [8, 9],
|
||||
WrapNonce: [10, 11],
|
||||
WrapTag: [12, 13],
|
||||
KekId: "kek-1",
|
||||
Revision: rev,
|
||||
CreatedUtc: DateTimeOffset.Parse("2026-01-01T00:00:00Z"),
|
||||
UpdatedUtc: DateTimeOffset.Parse("2026-01-02T00:00:00Z"));
|
||||
|
||||
private static DeployArtifactsCommand Cmd(
|
||||
IReadOnlyList<SecretArtifact>? secrets = null,
|
||||
IReadOnlyList<string>? missing = null) =>
|
||||
Command() with { Secrets = secrets, MissingSecrets = missing };
|
||||
|
||||
[Fact]
|
||||
public void SecretChanges_MoveTheHash_ButOrderDoesNot()
|
||||
{
|
||||
var svc = new ArtifactRevisionHashService();
|
||||
var a = Secret("A", rev: 1);
|
||||
var b = Secret("B", rev: 1);
|
||||
var baseline = svc.ComputeHash(Cmd(secrets: [a, b]));
|
||||
|
||||
Assert.Equal(baseline, svc.ComputeHash(Cmd(secrets: [b, a])));
|
||||
Assert.NotEqual(baseline, svc.ComputeHash(Cmd(secrets: [a with { Revision = 2 }, b])));
|
||||
Assert.NotEqual(baseline, svc.ComputeHash(Cmd(secrets: [a with { WrappedDek = [42] }, b])));
|
||||
Assert.NotEqual(baseline, svc.ComputeHash(Cmd(secrets: [a with { KekId = "kek-B" }, b])));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void MissingSecret_HashesDifferentlyFromPresent()
|
||||
{
|
||||
var svc = new ArtifactRevisionHashService();
|
||||
|
||||
Assert.NotEqual(
|
||||
svc.ComputeHash(Cmd(secrets: [Secret("A", 1)])),
|
||||
svc.ComputeHash(Cmd(missing: ["A"])));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ComputeHash_PreSecretsGoldenValue_IsUnchanged()
|
||||
{
|
||||
// Captured on HEAD 738e4587 — BEFORE Task 4 wired Secrets/MissingSecrets into the
|
||||
// projection — using the fixed default baseline (one of each existing artifact
|
||||
// kind) with a pinned deploymentId/timestamp. A command with Secrets == null and
|
||||
// MissingSecrets == null must keep hashing to this exact value after Task 4, or a
|
||||
// site with no secret references would spuriously flip to "out of date".
|
||||
var golden = Command(
|
||||
deploymentId: "golden",
|
||||
timestamp: DateTimeOffset.Parse("2026-09-28T00:00:00Z"));
|
||||
|
||||
Assert.Equal(
|
||||
"sha256:949f2c9c6d80432f94e9c2ad2a669b020a33c4a69653cf1b7b2b0aa217a467b0",
|
||||
_sut.ComputeHash(golden));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ComputeHash_SameInputTwice_ProducesSameHash()
|
||||
{
|
||||
|
||||
@@ -16,6 +16,7 @@ using ZB.MOM.WW.ScadaBridge.Communication;
|
||||
using ZB.MOM.WW.ScadaBridge.DeploymentManager.PendingUpdates;
|
||||
using ZB.MOM.WW.ScadaBridge.TemplateEngine.Flattening;
|
||||
using ZB.MOM.WW.ScadaBridge.TemplateEngine.Validation;
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests;
|
||||
|
||||
@@ -66,7 +67,10 @@ public class PendingUpdatesEvaluatorTests
|
||||
comms, _audit,
|
||||
Options.Create(new DeploymentManagerOptions()),
|
||||
NullLogger<ArtifactDeploymentService>.Instance,
|
||||
_artifactHasher);
|
||||
_artifactHasher,
|
||||
// None of these tests reference secrets, so a store that never has anything is
|
||||
// never exercised — CollectAsync only calls GetAsync for a scanned secret: reference.
|
||||
new SiteSecretsCollector(Substitute.For<ISecretStore>()));
|
||||
|
||||
_deploymentService = new DeploymentService(
|
||||
_deploymentRepo,
|
||||
|
||||
@@ -0,0 +1,136 @@
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// <see cref="SiteSecretsCollector"/> is the central half of design §3.1/§3.3: derive which
|
||||
/// secrets a site's data connections reference, resolve each against the central
|
||||
/// <see cref="ISecretStore"/>, and report the ones that could not be resolved. It never sees
|
||||
/// or produces plaintext — everything here is ciphertext bytes.
|
||||
/// </summary>
|
||||
public class SiteSecretsCollectorTests
|
||||
{
|
||||
private static DataConnectionArtifact Dc(string name, string? primary, string? backup = null)
|
||||
=> new(name, "MxGateway", primary, backup, 3);
|
||||
|
||||
private static StoredSecret Row(string name, long revision, bool deleted = false, string? description = null) => new()
|
||||
{
|
||||
Name = new SecretName(name),
|
||||
Description = description,
|
||||
ContentType = SecretContentType.Text,
|
||||
Ciphertext = [1, 2, 3],
|
||||
Nonce = [4, 5],
|
||||
Tag = [6, 7],
|
||||
WrappedDek = [8, 9],
|
||||
WrapNonce = [10, 11],
|
||||
WrapTag = [12, 13],
|
||||
KekId = "kek-1",
|
||||
Revision = revision,
|
||||
IsDeleted = deleted,
|
||||
CreatedUtc = DateTimeOffset.Parse("2026-01-01T00:00:00Z"),
|
||||
UpdatedUtc = DateTimeOffset.Parse("2026-01-02T00:00:00Z"),
|
||||
};
|
||||
|
||||
[Fact]
|
||||
public async Task Collect_ReturnsOnlyReferencedRows_AndNamesMissing()
|
||||
{
|
||||
var store = new FakeSecretStore(Row("a", 3), Row("b", 1, deleted: true), Row("other", 1));
|
||||
|
||||
var result = await new SiteSecretsCollector(store).CollectAsync(
|
||||
[Dc("GW1", """{"ApiKey":"secret:a"}"""), Dc("GW2", """{"ApiKey":"secret:b"}""")], default);
|
||||
|
||||
// Only the referenced names come back — "other" is in the store but never referenced.
|
||||
Assert.Equal(["a"], result.Secrets.Select(s => s.Name));
|
||||
Assert.Equal(3, result.Secrets[0].Revision);
|
||||
// "b" is tombstoned (IsDeleted) — treated the same as absent.
|
||||
Assert.Equal(["b"], result.Missing.Select(m => m.Name));
|
||||
Assert.Equal(["GW2"], result.Missing[0].ReferencedBy);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Collect_ResolvedRow_MapsEveryFieldFromTheStoredSecretVerbatim()
|
||||
{
|
||||
var row = Row("mxgw-key", revision: 7, description: "MxGateway API key");
|
||||
var store = new FakeSecretStore(row);
|
||||
|
||||
var result = await new SiteSecretsCollector(store).CollectAsync(
|
||||
[Dc("GW1", """{"ApiKey":"secret:mxgw-key"}""")], default);
|
||||
|
||||
var artifact = Assert.Single(result.Secrets);
|
||||
Assert.Empty(result.Missing);
|
||||
Assert.Equal(row.Name.Value, artifact.Name);
|
||||
Assert.Equal(row.ContentType.ToString(), artifact.ContentType);
|
||||
Assert.Equal(row.Ciphertext, artifact.Ciphertext);
|
||||
Assert.Equal(row.Nonce, artifact.Nonce);
|
||||
Assert.Equal(row.Tag, artifact.Tag);
|
||||
Assert.Equal(row.WrappedDek, artifact.WrappedDek);
|
||||
Assert.Equal(row.WrapNonce, artifact.WrapNonce);
|
||||
Assert.Equal(row.WrapTag, artifact.WrapTag);
|
||||
Assert.Equal(row.KekId, artifact.KekId);
|
||||
Assert.Equal(row.Revision, artifact.Revision);
|
||||
Assert.Equal(row.CreatedUtc, artifact.CreatedUtc);
|
||||
Assert.Equal(row.UpdatedUtc, artifact.UpdatedUtc);
|
||||
Assert.Equal(row.Description, artifact.Description);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Collect_AbsentName_IsReportedMissing_NotThrown()
|
||||
{
|
||||
var store = new FakeSecretStore();
|
||||
|
||||
var result = await new SiteSecretsCollector(store).CollectAsync(
|
||||
[Dc("GW1", """{"ApiKey":"secret:never-created"}""")], default);
|
||||
|
||||
Assert.Empty(result.Secrets);
|
||||
var missing = Assert.Single(result.Missing);
|
||||
Assert.Equal("never-created", missing.Name);
|
||||
Assert.Equal(["GW1"], missing.ReferencedBy);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Collect_NoSecretReferences_ReturnsEmptyWithoutQueryingTheStore()
|
||||
{
|
||||
var store = new FakeSecretStore(Row("unused", 1));
|
||||
|
||||
var result = await new SiteSecretsCollector(store).CollectAsync(
|
||||
[Dc("GW1", """{"Endpoint":"http://gw:5000"}""")], default);
|
||||
|
||||
Assert.Empty(result.Secrets);
|
||||
Assert.Empty(result.Missing);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// In-memory <see cref="ISecretStore"/> stand-in. Implements only <see cref="GetAsync"/> —
|
||||
/// the sole member <see cref="SiteSecretsCollector"/> calls; every other member throws so an
|
||||
/// accidental call (a write, a list, a replication apply) fails the test loudly instead of
|
||||
/// silently no-op'ing.
|
||||
/// </summary>
|
||||
private sealed class FakeSecretStore : ISecretStore
|
||||
{
|
||||
private readonly Dictionary<SecretName, StoredSecret> _rows;
|
||||
|
||||
public FakeSecretStore(params StoredSecret[] rows) => _rows = rows.ToDictionary(r => r.Name);
|
||||
|
||||
public Task<StoredSecret?> GetAsync(SecretName name, CancellationToken ct) =>
|
||||
Task.FromResult(_rows.TryGetValue(name, out var row) ? row : null);
|
||||
|
||||
public Task UpsertAsync(StoredSecret row, CancellationToken ct) => throw new NotSupportedException();
|
||||
|
||||
public Task<bool> DeleteAsync(SecretName name, string? actor, CancellationToken ct) =>
|
||||
throw new NotSupportedException();
|
||||
|
||||
public Task<IReadOnlyList<SecretMetadata>> ListAsync(bool includeDeleted, CancellationToken ct) =>
|
||||
throw new NotSupportedException();
|
||||
|
||||
public Task<IReadOnlyList<SecretManifestEntry>> GetManifestAsync(CancellationToken ct) =>
|
||||
throw new NotSupportedException();
|
||||
|
||||
public Task ApplyReplicatedAsync(StoredSecret row, CancellationToken ct) =>
|
||||
throw new NotSupportedException();
|
||||
|
||||
public Task<bool> ApplyRewrapAsync(
|
||||
StoredSecret rewrappedRow, byte[] expectedCurrentWrappedDek, CancellationToken ct) =>
|
||||
throw new NotSupportedException();
|
||||
}
|
||||
}
|
||||
@@ -5,12 +5,8 @@ using Microsoft.AspNetCore.Builder;
|
||||
using Microsoft.AspNetCore.Routing;
|
||||
using Microsoft.CodeAnalysis.CSharp;
|
||||
using Microsoft.CodeAnalysis.CSharp.Syntax;
|
||||
using Microsoft.Extensions.Configuration;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.LocalDb.Replication;
|
||||
using ZB.MOM.WW.Secrets.Replicator.Grpc;
|
||||
using ZB.MOM.WW.Secrets.Replicator.Grpc.DependencyInjection;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
|
||||
|
||||
@@ -21,14 +17,7 @@ public class MappedGrpcAuthenticationCoverageTests
|
||||
public async Task EveryMappedGrpcService_IsCoveredIncludingExternalMappings()
|
||||
{
|
||||
var builder = WebApplication.CreateBuilder();
|
||||
builder.Configuration.AddInMemoryCollection(new Dictionary<string, string?>
|
||||
{
|
||||
["Secrets:Replication:Enabled"] = "true",
|
||||
["Secrets:Replication:Mode"] = "Grpc",
|
||||
["Secrets:GrpcHub:BearerToken"] = "coverage-test-token",
|
||||
});
|
||||
builder.Services.AddGrpc();
|
||||
builder.Services.AddZbSecretsGrpcHub(builder.Configuration, "Secrets:GrpcHub");
|
||||
await using var app = builder.Build();
|
||||
|
||||
// Discover the production map calls, not a hand-maintained three-service type list.
|
||||
@@ -51,17 +40,19 @@ public class MappedGrpcAuthenticationCoverageTests
|
||||
map.MakeGenericMethod(type).Invoke(null, [app]);
|
||||
}
|
||||
|
||||
// Exercise the production extension mappings too; these packages hide their MapGrpcService calls.
|
||||
// Exercise the production extension mappings too; this package hides its MapGrpcService call.
|
||||
Assert.Contains(hostSources.SelectMany(s => s.DescendantNodes().OfType<InvocationExpressionSyntax>()),
|
||||
n => n.Expression.ToString().EndsWith(".MapZbLocalDbSync", StringComparison.Ordinal));
|
||||
app.MapZbLocalDbSync();
|
||||
app.MapScadaBridgeSecretsHub(builder.Configuration);
|
||||
var metadata = ((IEndpointRouteBuilder)app).DataSources.SelectMany(s => s.Endpoints)
|
||||
.Select(e => e.Metadata.GetMetadata<GrpcMethodMetadata>()).Where(m => m is not null).ToList();
|
||||
Assert.NotEmpty(metadata);
|
||||
var serviceTypes = metadata.Select(m => m!.ServiceType).Distinct().ToList();
|
||||
Assert.True(serviceTypes.Count >= typeNames.Count + 2, "External LocalDb and Secrets mappings must both be exercised.");
|
||||
Assert.True(serviceTypes.Count >= typeNames.Count + 1, "The external LocalDb mapping must also be exercised.");
|
||||
|
||||
// The secrets hub — which used to install an all-method interceptor on its own service
|
||||
// options and so needed a separate gate check here — is retired (Task 11): there is no
|
||||
// longer a mapped gRPC service whose only gate is anything other than one of these prefixes.
|
||||
var prefixes = ControlPlaneAuthInterceptor.DefaultGatedPrefixes
|
||||
.Concat(CentralControlAuthInterceptor.DefaultGatedPrefixes)
|
||||
.Append((string)typeof(LocalDbSyncAuthInterceptor).GetField("ServicePrefix", BindingFlags.Static | BindingFlags.NonPublic)!.GetRawConstantValue()!)
|
||||
@@ -78,21 +69,14 @@ public class MappedGrpcAuthenticationCoverageTests
|
||||
Assert.NotEmpty(binder.Methods);
|
||||
Assert.Equal(metadata.Where(m => m!.ServiceType == type).Select(m => m!.Method.FullName).Order(), binder.Methods.Order());
|
||||
|
||||
// Secrets installs an all-method interceptor on its own service options. Inspect actual DI options.
|
||||
var optionsType = typeof(IOptions<>).MakeGenericType(typeof(GrpcServiceOptions<>).MakeGenericType(type));
|
||||
var options = app.Services.GetRequiredService(optionsType);
|
||||
var serviceOptions = (GrpcServiceOptions)optionsType.GetProperty("Value")!.GetValue(options)!;
|
||||
var hasSecretsGate = serviceOptions.Interceptors.Any(i => i.Type == typeof(SecretsHubAuthInterceptor));
|
||||
Assert.All(binder.Methods, method => Assert.True(
|
||||
hasSecretsGate || prefixes.Any(prefix => method.StartsWith(prefix, StringComparison.Ordinal)),
|
||||
prefixes.Any(prefix => method.StartsWith(prefix, StringComparison.Ordinal)),
|
||||
$"Mapped service {type.FullName} method {method} has no authentication interceptor gate."));
|
||||
if (!hasSecretsGate)
|
||||
{
|
||||
// Negative control uses the real service's methods: removing its gate must be detected.
|
||||
var removed = prefixes.First(p => binder.Methods.Any(m => m.StartsWith(p, StringComparison.Ordinal)));
|
||||
var remaining = prefixes.Where(p => p != removed).ToArray();
|
||||
Assert.Contains(binder.Methods, method => !remaining.Any(p => method.StartsWith(p, StringComparison.Ordinal)));
|
||||
}
|
||||
|
||||
// Negative control uses the real service's methods: removing its gate must be detected.
|
||||
var removed = prefixes.First(p => binder.Methods.Any(m => m.StartsWith(p, StringComparison.Ordinal)));
|
||||
var remaining = prefixes.Where(p => p != removed).ToArray();
|
||||
Assert.Contains(binder.Methods, method => !remaining.Any(p => method.StartsWith(p, StringComparison.Ordinal)));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -1,13 +1,6 @@
|
||||
using Grpc.Net.Client;
|
||||
using Microsoft.AspNetCore.Builder;
|
||||
using Microsoft.AspNetCore.Routing;
|
||||
using Microsoft.Extensions.Configuration;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
using ZB.MOM.WW.Secrets.Replication;
|
||||
using ZB.MOM.WW.Secrets.Replicator.Grpc;
|
||||
using ZB.MOM.WW.Secrets.Replicator.Grpc.DependencyInjection;
|
||||
using ZB.MOM.WW.Secrets.Replicator.SqlServer;
|
||||
using ZB.MOM.WW.Secrets.Sqlite;
|
||||
|
||||
@@ -16,6 +9,14 @@ namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
|
||||
/// <summary>
|
||||
/// DI-resolution tests for <see cref="SecretsRegistration.AddScadaBridgeSecrets"/>.
|
||||
/// <para>
|
||||
/// Task 11 retired every secrets REPLICATION mode in ScadaBridge — the pull-only gRPC hub (central
|
||||
/// hosts, sites sweep) and the SQL-Server <c>ReplicatingSecretStore</c> mode that preceded it — on
|
||||
/// BOTH roles. What is left: the plain local SQLite store, which is every node's default and now a
|
||||
/// Site's ONLY possible shape, and the still-supported Task 5
|
||||
/// <see cref="SecretsRegistration.StoreProviderKey"/>=SqlServer direct shared store (Central-only,
|
||||
/// no hub, no follower sweep — just two nodes reading and writing one database).
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// These deliberately BUILD a container and RESOLVE out of it rather than asserting over
|
||||
/// <see cref="ServiceDescriptor"/>s. A decorator registration can look perfectly correct as a
|
||||
/// descriptor list and still throw on the first resolve because the undecorated concrete store it
|
||||
@@ -25,10 +26,8 @@ namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
|
||||
/// <para>
|
||||
/// Registration must not need a reachable SQL Server: the SQL-Server package validates its options
|
||||
/// eagerly, but validation only checks that the connection string is non-empty and the schema name
|
||||
/// is a legal identifier. Nothing connects until a sweep or a write runs, and these tests do
|
||||
/// neither — so the dummy connection string below never resolves to a real host. The same holds for
|
||||
/// the gRPC hub client: <c>GrpcChannel.ForAddress</c> performs no I/O, so the dummy endpoint is
|
||||
/// never dialled either.
|
||||
/// is a legal identifier. Nothing connects until a write or read runs, and these tests do neither —
|
||||
/// so the dummy connection string below never resolves to a real host.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public class SecretsReplicationWiringTests
|
||||
@@ -37,32 +36,16 @@ public class SecretsReplicationWiringTests
|
||||
private const string DummyConnectionString =
|
||||
"Server=unused;Database=x;Integrated Security=true;";
|
||||
|
||||
// Syntactically valid, deliberately unreachable. Never dialled by these tests.
|
||||
private const string DummyHubEndpoint = "http://unused.invalid:8083";
|
||||
|
||||
// Syntactically valid, deliberately unreachable. Never dialled by these tests.
|
||||
private const string DummyFallbackHubEndpoint = "http://unused-fallback.invalid:8083";
|
||||
|
||||
/// <summary>
|
||||
/// Keyed service key of the <see cref="GrpcChannel"/> dialing fallback endpoint 0 — the 0.5.0
|
||||
/// package contract: the primary channel keeps
|
||||
/// <see cref="SecretsGrpcHubClientExtensions.ChannelServiceKey"/> unchanged and fallback
|
||||
/// <c>i</c> gets <c>:fallback:i</c> appended.
|
||||
/// A fresh directory, never the one holding <see cref="BuildConfig"/>'s default
|
||||
/// <c>Secrets:SqlitePath</c> (<c>Path.GetTempPath()</c> itself) — the SQLite path-rules
|
||||
/// validator runs eagerly on the first <c>IOptions<SecretsOptions>.Value</c> read (i.e.
|
||||
/// on resolving <c>ISecretStore</c> itself, not merely on migrating), so passing
|
||||
/// <c>Path.GetTempPath()</c> as the content root here would make every SQLite-path expander
|
||||
/// test fail on "path is inside the content root" instead of testing what it means to.
|
||||
/// </summary>
|
||||
private const string FallbackChannelKey =
|
||||
SecretsGrpcHubClientExtensions.ChannelServiceKey + ":fallback:0";
|
||||
|
||||
/// <summary>
|
||||
/// The package-internal reader seam the follower sweep pulls through
|
||||
/// (<c>ISecretsHubReader</c>). Internal to the package, so it is obtained by full name and
|
||||
/// resolved by <see cref="Type"/> — visibility never gates DI resolution, only compile-time
|
||||
/// references.
|
||||
/// </summary>
|
||||
private static readonly Type HubReaderInterface = typeof(SecretsHubAuthInterceptor).Assembly
|
||||
.GetType("ZB.MOM.WW.Secrets.Replicator.Grpc.ISecretsHubReader", throwOnError: true)!;
|
||||
|
||||
// Not a credential — a non-blank placeholder, which is all the fail-closed validators check.
|
||||
private const string DummyBearerToken = "test-hub-token";
|
||||
private static string NewContentRoot() =>
|
||||
Path.Combine(Path.GetTempPath(), "sb-secrets-wiring-content-root-" + Guid.NewGuid().ToString("N"));
|
||||
|
||||
private static IConfiguration BuildConfig(params (string Key, string Value)[] overrides)
|
||||
{
|
||||
@@ -93,377 +76,143 @@ public class SecretsReplicationWiringTests
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Any hosted service contributed by the SQL-Server replicator package. Asserted by assembly
|
||||
/// rather than by type because <c>SqlServerHubMigrationHostedService</c> is internal to it.
|
||||
/// </summary>
|
||||
private static bool IsSqlServerReplicatorHostedService(ServiceDescriptor descriptor) =>
|
||||
descriptor.ServiceType == typeof(IHostedService)
|
||||
&& descriptor.ImplementationType?.Assembly.GetName().Name
|
||||
== "ZB.MOM.WW.Secrets.Replicator.SqlServer";
|
||||
|
||||
/// <summary>
|
||||
/// Any descriptor whose service OR implementation type lives in the SQL-Server replicator
|
||||
/// assembly. Honest bound: the package also registers <c>ISecretStore</c>/<c>ISecretsStoreMigrator</c>
|
||||
/// Any descriptor whose service OR implementation type lives in the SQL-Server package.
|
||||
/// Honest bound: the package also registers <c>ISecretStore</c>/<c>ISecretsStoreMigrator</c>
|
||||
/// through factory lambdas whose service type is the Abstractions interface and whose
|
||||
/// ImplementationType is null — those two descriptors would evade this scan in isolation. The
|
||||
/// site-purity pin still holds because the same extension unconditionally registers three
|
||||
/// concrete types from the target assembly first, which this scan does catch — so the package
|
||||
/// cannot enter the container without tripping it, even though not every individual descriptor
|
||||
/// it adds is individually detectable.
|
||||
/// site-purity pin still holds because the same extension unconditionally registers concrete
|
||||
/// types from the target assembly first, which this scan does catch — so the package cannot
|
||||
/// enter the container without tripping it, even though not every individual descriptor it
|
||||
/// adds is individually detectable.
|
||||
/// </summary>
|
||||
private static bool IsFromSqlServerReplicatorPackage(ServiceDescriptor descriptor) =>
|
||||
private static bool IsFromSqlServerPackage(ServiceDescriptor descriptor) =>
|
||||
descriptor.ServiceType.Assembly.GetName().Name == "ZB.MOM.WW.Secrets.Replicator.SqlServer"
|
||||
|| descriptor.ImplementationType?.Assembly.GetName().Name
|
||||
== "ZB.MOM.WW.Secrets.Replicator.SqlServer";
|
||||
|
||||
/// <summary>
|
||||
/// The gRPC hub package's one hosted service — the follower sweep. Asserted by assembly and
|
||||
/// type NAME because <c>GrpcSecretSyncService</c> is internal to the package. The hub SERVER
|
||||
/// half contributes no hosted service at all, so presence of one from this assembly is
|
||||
/// specifically the follower half and nothing else.
|
||||
/// </summary>
|
||||
private static bool IsGrpcHubSweepService(IHostedService service) =>
|
||||
service.GetType().Assembly.GetName().Name == "ZB.MOM.WW.Secrets.Replicator.Grpc"
|
||||
&& service.GetType().Name == "GrpcSecretSyncService";
|
||||
|
||||
/// <summary>
|
||||
/// The complete central shape for gRPC mode, connection string included: central's store in
|
||||
/// this mode is the SHARED SQL-Server store (scadaproj#4), so a central config without a
|
||||
/// connection string is not a valid variant — it is a registration failure, pinned separately.
|
||||
/// </summary>
|
||||
private static (string Key, string Value)[] GrpcCentralConfig() =>
|
||||
[
|
||||
("Secrets:Replication:Enabled", "true"),
|
||||
("Secrets:Replication:Mode", "Grpc"),
|
||||
("Secrets:GrpcHub:BearerToken", DummyBearerToken),
|
||||
("Secrets:SqlServer:ConnectionString", DummyConnectionString),
|
||||
];
|
||||
|
||||
private static (string Key, string Value)[] GrpcSiteConfig() =>
|
||||
[
|
||||
("Secrets:Replication:Enabled", "true"),
|
||||
("Secrets:Replication:Mode", "Grpc"),
|
||||
("Secrets:GrpcHub:Endpoint", DummyHubEndpoint),
|
||||
("Secrets:GrpcHub:BearerToken", DummyBearerToken),
|
||||
];
|
||||
|
||||
// ---------------------------------------------------------------------------------------
|
||||
// Replication off — the default, and the shape every existing deployment is in.
|
||||
// Site: the ONLY shape left, on every configuration — plain local SQLite, never a
|
||||
// replication decorator (Task 11, ruling #1). DeployedSecretsMirror deletes a name from
|
||||
// this store directly the moment a deploy artifact stops referencing it; a decorator would
|
||||
// publish that delete to every other node sharing whatever it decorates against, turning
|
||||
// one site's unreferencing of a secret into a fleet-wide deletion.
|
||||
// ---------------------------------------------------------------------------------------
|
||||
|
||||
[Fact]
|
||||
public void ReplicationDisabledByDefault_ResolvesPlainSqliteStore()
|
||||
public void Site_ResolvesPlainSqliteStore_NeverAReplicationDecorator()
|
||||
{
|
||||
using ServiceProvider provider = BuildProvider(BuildConfig());
|
||||
using ServiceProvider provider = BuildProvider(BuildConfig(), SecretsNodeRole.Site);
|
||||
|
||||
var store = provider.GetRequiredService<ISecretStore>();
|
||||
|
||||
// Assert.IsType checks the EXACT runtime type, not merely "is a SqliteSecretStore or a
|
||||
// decorator wrapping one" — nothing may sit between the site and its local file.
|
||||
Assert.IsType<SqliteSecretStore>(store);
|
||||
Assert.IsNotType<ReplicatingSecretStore>(store);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ReplicationDisabledByDefault_RegistersNoReplicatorHostedServices()
|
||||
public void Site_IgnoresStoreProvider_EvenSetToSqlServer()
|
||||
{
|
||||
var services = new ServiceCollection();
|
||||
services.AddLogging();
|
||||
services.AddScadaBridgeSecrets(BuildConfig(), SecretsNodeRole.Central);
|
||||
// A Site node ignores Secrets:Store:Provider entirely and always keeps its local SQLite
|
||||
// store — even with the value set to SqlServer and no connection string configured at
|
||||
// all, which would fail closed on Central.
|
||||
IConfiguration config = BuildConfig(("Secrets:Store:Provider", "SqlServer"));
|
||||
|
||||
Assert.DoesNotContain(services, IsSqlServerReplicatorHostedService);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Flag off ⇒ NEITHER half of the gRPC hub, on EITHER role — even with the hub section fully
|
||||
/// populated. The mode key selects a transport; it must never be what turns replication on.
|
||||
/// </summary>
|
||||
[Theory]
|
||||
[InlineData(SecretsNodeRole.Central)]
|
||||
[InlineData(SecretsNodeRole.Site)]
|
||||
public void ReplicationDisabled_WithGrpcModeAndFullHubConfig_WiresNeitherHubHalf(
|
||||
SecretsNodeRole role)
|
||||
{
|
||||
IConfiguration config = BuildConfig(
|
||||
("Secrets:Replication:Enabled", "false"),
|
||||
("Secrets:Replication:Mode", "Grpc"),
|
||||
("Secrets:GrpcHub:Endpoint", DummyHubEndpoint),
|
||||
("Secrets:GrpcHub:BearerToken", DummyBearerToken));
|
||||
|
||||
using ServiceProvider provider = BuildProvider(config, role);
|
||||
using ServiceProvider provider = BuildProvider(config, SecretsNodeRole.Site);
|
||||
|
||||
Assert.IsType<SqliteSecretStore>(provider.GetRequiredService<ISecretStore>());
|
||||
Assert.Null(provider.GetService<SecretsHubAuthInterceptor>());
|
||||
Assert.DoesNotContain(provider.GetServices<IHostedService>(), IsGrpcHubSweepService);
|
||||
Assert.False(SecretsRegistration.UsesGrpcHub(config));
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------------------
|
||||
// SQL-Server mode — regression pins. Behaviour must be byte-for-byte what it was before the
|
||||
// mode key existed, both when the mode is left unset and when it is named explicitly.
|
||||
// ---------------------------------------------------------------------------------------
|
||||
|
||||
[Fact]
|
||||
public void ReplicationEnabledWithoutConnectionString_StaysOnPlainSqliteStore()
|
||||
public void Site_HasNoSqlServerPackageTypesInTheContainer()
|
||||
{
|
||||
// The gate requires BOTH the flag and a connection string. Enabling the flag alone must not
|
||||
// reach the SQL-Server package, whose options validation would throw at registration and
|
||||
// take the whole node down at startup.
|
||||
// The sites-don't-talk-to-SQL pin: a site node's container must contain NOTHING from the
|
||||
// SQL-Server package — not just resolve a SQLite store, but hold no descriptor that could
|
||||
// ever open a connection to central's database.
|
||||
var services = new ServiceCollection();
|
||||
services.AddLogging();
|
||||
services.AddScadaBridgeSecrets(BuildConfig(), SecretsNodeRole.Site);
|
||||
|
||||
IConfiguration config = BuildConfig(("Secrets:Replication:Enabled", "true"));
|
||||
|
||||
services.AddScadaBridgeSecrets(config, SecretsNodeRole.Central);
|
||||
Assert.DoesNotContain(services, IsFromSqlServerPackage);
|
||||
|
||||
using ServiceProvider provider = services.BuildServiceProvider(validateScopes: true);
|
||||
|
||||
Assert.IsType<SqliteSecretStore>(provider.GetRequiredService<ISecretStore>());
|
||||
Assert.DoesNotContain(services, IsSqlServerReplicatorHostedService);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(null)]
|
||||
[InlineData("SqlServer")]
|
||||
public void ReplicationEnabledWithConnectionString_ResolvesReplicatingStore(string? mode)
|
||||
{
|
||||
IConfiguration config = mode is null
|
||||
? BuildConfig(
|
||||
("Secrets:Replication:Enabled", "true"),
|
||||
("Secrets:SqlServer:ConnectionString", DummyConnectionString))
|
||||
: BuildConfig(
|
||||
("Secrets:Replication:Enabled", "true"),
|
||||
("Secrets:Replication:Mode", mode),
|
||||
("Secrets:SqlServer:ConnectionString", DummyConnectionString));
|
||||
|
||||
using ServiceProvider provider = BuildProvider(config);
|
||||
|
||||
Assert.IsType<ReplicatingSecretStore>(provider.GetRequiredService<ISecretStore>());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ReplicationEnabledWithConnectionString_UndecoratedLocalStoreAlsoResolves()
|
||||
{
|
||||
// ReplicatingSecretStore is constructed from the CONCRETE SqliteSecretStore, not from
|
||||
// ISecretStore (which is the decorator itself — that would recurse). If the concrete
|
||||
// registration were ever dropped, the decorator would fail on first resolve. This is the
|
||||
// exact defect that shipped once, so it gets its own test.
|
||||
IConfiguration config = BuildConfig(
|
||||
("Secrets:Replication:Enabled", "true"),
|
||||
("Secrets:SqlServer:ConnectionString", DummyConnectionString));
|
||||
|
||||
using ServiceProvider provider = BuildProvider(config);
|
||||
|
||||
Assert.NotNull(provider.GetRequiredService<SqliteSecretStore>());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ReplicationEnabledWithConnectionString_RegistersReplicatorHostedServices()
|
||||
{
|
||||
var services = new ServiceCollection();
|
||||
services.AddLogging();
|
||||
|
||||
IConfiguration config = BuildConfig(
|
||||
("Secrets:Replication:Enabled", "true"),
|
||||
("Secrets:SqlServer:ConnectionString", DummyConnectionString));
|
||||
|
||||
services.AddScadaBridgeSecrets(config, SecretsNodeRole.Central);
|
||||
|
||||
// Both the hub-schema migration service and the bidirectional sweep.
|
||||
Assert.Equal(2, services.Count(IsSqlServerReplicatorHostedService));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// SQL-Server mode is role-agnostic — both nodes of a hub topology sync bidirectionally
|
||||
/// against the same shared database. The role parameter added for the gRPC hub must not have
|
||||
/// quietly changed that.
|
||||
/// </summary>
|
||||
[Theory]
|
||||
[InlineData(SecretsNodeRole.Central)]
|
||||
[InlineData(SecretsNodeRole.Site)]
|
||||
public void SqlServerMode_IsIdenticalOnBothRoles(SecretsNodeRole role)
|
||||
{
|
||||
IConfiguration config = BuildConfig(
|
||||
("Secrets:Replication:Enabled", "true"),
|
||||
("Secrets:Replication:Mode", "SqlServer"),
|
||||
("Secrets:SqlServer:ConnectionString", DummyConnectionString));
|
||||
|
||||
using ServiceProvider provider = BuildProvider(config, role);
|
||||
|
||||
Assert.IsType<ReplicatingSecretStore>(provider.GetRequiredService<ISecretStore>());
|
||||
Assert.Null(provider.GetService<SecretsHubAuthInterceptor>());
|
||||
Assert.DoesNotContain(provider.GetServices<IHostedService>(), IsGrpcHubSweepService);
|
||||
Assert.Null(provider.GetService<SqlServerSecretStore>());
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------------------
|
||||
// gRPC mode — central hosts, site follows, and neither does the other's job.
|
||||
// Central, default — plain local SQLite, the shape every existing deployment is in.
|
||||
// ---------------------------------------------------------------------------------------
|
||||
|
||||
[Fact]
|
||||
public void GrpcMode_Central_RegistersHubAndItsAuthInterceptor()
|
||||
public void Central_DefaultsToPlainSqliteStore()
|
||||
{
|
||||
IConfiguration config = BuildConfig(GrpcCentralConfig());
|
||||
using ServiceProvider provider = BuildProvider(BuildConfig());
|
||||
|
||||
using ServiceProvider provider = BuildProvider(config, SecretsNodeRole.Central);
|
||||
|
||||
// The fail-closed bearer gate must RESOLVE, not merely be listed: an interceptor that
|
||||
// cannot be activated is one gRPC would fail to attach, which is an ungated hub.
|
||||
Assert.NotNull(provider.GetService<SecretsHubAuthInterceptor>());
|
||||
Assert.True(SecretsRegistration.UsesGrpcHub(config));
|
||||
|
||||
// Central's store is the SHARED SQL-Server store — one copy of every row, so both
|
||||
// central hubs serve identical manifests by construction (scadaproj#4). Undecorated:
|
||||
// pull-only means nothing publishes, so there is still nothing to wrap it in.
|
||||
Assert.IsType<SqlServerSecretStore>(provider.GetRequiredService<ISecretStore>());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void GrpcMode_Central_DoesNotRegisterTheFollowerSweep()
|
||||
{
|
||||
// Central is the origin of every write. A sweep here would have it pull from itself, and
|
||||
// on a misconfigured endpoint, from somebody else.
|
||||
using ServiceProvider provider =
|
||||
BuildProvider(BuildConfig(GrpcCentralConfig()), SecretsNodeRole.Central);
|
||||
|
||||
Assert.DoesNotContain(provider.GetServices<IHostedService>(), IsGrpcHubSweepService);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void GrpcMode_Site_RegistersTheFollowerSweep()
|
||||
{
|
||||
using ServiceProvider provider =
|
||||
BuildProvider(BuildConfig(GrpcSiteConfig()), SecretsNodeRole.Site);
|
||||
|
||||
// Resolving the hosted services builds the sweep's whole graph — reader, keyed channel,
|
||||
// local store — which is where a missing AddZbSecrets would surface.
|
||||
Assert.Contains(provider.GetServices<IHostedService>(), IsGrpcHubSweepService);
|
||||
Assert.IsType<SqliteSecretStore>(provider.GetRequiredService<ISecretStore>());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void GrpcMode_Site_DoesNotHostTheHub()
|
||||
public void Central_StoreProviderSqlite_IsIndistinguishableFromUnset()
|
||||
{
|
||||
// A site hosting the hub would serve central's secrets to anything holding the shared
|
||||
// token, from inside the site network.
|
||||
using ServiceProvider provider =
|
||||
BuildProvider(BuildConfig(GrpcSiteConfig()), SecretsNodeRole.Site);
|
||||
using ServiceProvider explicitProvider =
|
||||
BuildProvider(BuildConfig(("Secrets:Store:Provider", "Sqlite")));
|
||||
using ServiceProvider unsetProvider = BuildProvider(BuildConfig());
|
||||
|
||||
Assert.Null(provider.GetService<SecretsHubAuthInterceptor>());
|
||||
Assert.IsType<SqliteSecretStore>(explicitProvider.GetRequiredService<ISecretStore>());
|
||||
Assert.IsType<SqliteSecretStore>(unsetProvider.GetRequiredService<ISecretStore>());
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A configured fallback endpoint swaps the sweep's reader for the package's failover
|
||||
/// composition and adds one keyed channel per fallback. Safe ONLY because both central nodes
|
||||
/// serve one shared SQL store (scadaproj#4) — the type-name assertion is deliberate: the
|
||||
/// failover reader is internal to the package, and its name is the observable contract here.
|
||||
/// </summary>
|
||||
// ---------------------------------------------------------------------------------------
|
||||
// Secrets:Store:Provider=SqlServer — a clustered central sharing ONE SQL secret store
|
||||
// directly, without a hub (task 5). The one replication-adjacent feature Task 11 left
|
||||
// standing, because it is a plain shared database, not a converging replica set.
|
||||
// ---------------------------------------------------------------------------------------
|
||||
|
||||
[Fact]
|
||||
public void GrpcMode_Site_WithFallbackEndpoint_ResolvesTheFailoverReader()
|
||||
public void Central_StoreProviderSqlServer_UsesSharedSqlStore()
|
||||
{
|
||||
IConfiguration config = BuildConfig(
|
||||
[
|
||||
.. GrpcSiteConfig(),
|
||||
("Secrets:GrpcHub:FallbackEndpoints:0", DummyFallbackHubEndpoint),
|
||||
]);
|
||||
("Secrets:Store:Provider", "SqlServer"),
|
||||
("Secrets:SqlServer:ConnectionString", DummyConnectionString));
|
||||
|
||||
using ServiceProvider provider = BuildProvider(config, SecretsNodeRole.Site);
|
||||
using ServiceProvider provider = BuildProvider(config, SecretsNodeRole.Central);
|
||||
|
||||
object reader = provider.GetRequiredService(HubReaderInterface);
|
||||
Assert.Equal("FailoverSecretsHubReader", reader.GetType().Name);
|
||||
Assert.NotNull(provider.GetKeyedService<GrpcChannel>(FallbackChannelKey));
|
||||
Assert.IsType<SqlServerSecretStore>(provider.GetRequiredService<ISecretStore>());
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Zero fallbacks pins the unchanged default: the reader stays the plain single-endpoint
|
||||
/// client and no fallback channel enters the container — exactly the pre-0.5.0 shape every
|
||||
/// existing deployment is in.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void GrpcMode_Site_WithoutFallbackEndpoints_KeepsThePlainHubClient()
|
||||
{
|
||||
using ServiceProvider provider =
|
||||
BuildProvider(BuildConfig(GrpcSiteConfig()), SecretsNodeRole.Site);
|
||||
|
||||
object reader = provider.GetRequiredService(HubReaderInterface);
|
||||
Assert.Equal("GrpcSecretsHubClient", reader.GetType().Name);
|
||||
Assert.Null(provider.GetKeyedService<GrpcChannel>(FallbackChannelKey));
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------------------
|
||||
// gRPC mode fails closed. There is no local-only fallback here, unlike SQL-Server mode:
|
||||
// a site quietly serving secrets that never converge is the outcome the hub exists to prevent.
|
||||
// ---------------------------------------------------------------------------------------
|
||||
|
||||
[Fact]
|
||||
public void GrpcMode_Central_WithoutBearerToken_FailsAtRegistration()
|
||||
public void Central_StoreProviderSqlServer_BlankConnectionString_Throws()
|
||||
{
|
||||
var services = new ServiceCollection();
|
||||
services.AddLogging();
|
||||
|
||||
// Connection string present so the missing bearer token is the ONE defect in this config —
|
||||
// the connection-string pre-check runs first and would otherwise mask it.
|
||||
IConfiguration config = BuildConfig(
|
||||
("Secrets:Replication:Enabled", "true"),
|
||||
("Secrets:Replication:Mode", "Grpc"),
|
||||
("Secrets:SqlServer:ConnectionString", DummyConnectionString));
|
||||
IConfiguration config = BuildConfig(("Secrets:Store:Provider", "SqlServer"));
|
||||
|
||||
var ex = Assert.Throws<InvalidOperationException>(
|
||||
() => services.AddScadaBridgeSecrets(config, SecretsNodeRole.Central));
|
||||
|
||||
// The package's own message, unwrapped, naming the exact key an operator has to set.
|
||||
Assert.Contains("Secrets:GrpcHub:BearerToken", ex.Message, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Central + gRPC + no connection string is the scadaproj#4 defect asked for by name: each
|
||||
/// central node on an independent local store, one of them answering authenticated followers
|
||||
/// with an EMPTY manifest. It must be a boot failure naming the key, never a quiet fallback.
|
||||
/// </summary>
|
||||
[Theory]
|
||||
[InlineData(null)]
|
||||
[InlineData("")]
|
||||
[InlineData(" ")]
|
||||
public void GrpcMode_Central_WithBlankConnectionString_FailsAtRegistration(string? connectionString)
|
||||
{
|
||||
var services = new ServiceCollection();
|
||||
services.AddLogging();
|
||||
|
||||
(string, string)[] overrides = connectionString is null
|
||||
?
|
||||
[
|
||||
("Secrets:Replication:Enabled", "true"),
|
||||
("Secrets:Replication:Mode", "Grpc"),
|
||||
("Secrets:GrpcHub:BearerToken", DummyBearerToken),
|
||||
]
|
||||
:
|
||||
[
|
||||
("Secrets:Replication:Enabled", "true"),
|
||||
("Secrets:Replication:Mode", "Grpc"),
|
||||
("Secrets:GrpcHub:BearerToken", DummyBearerToken),
|
||||
("Secrets:SqlServer:ConnectionString", connectionString),
|
||||
];
|
||||
|
||||
var ex = Assert.Throws<InvalidOperationException>(
|
||||
() => services.AddScadaBridgeSecrets(BuildConfig(overrides), SecretsNodeRole.Central));
|
||||
|
||||
Assert.Contains(
|
||||
SecretsRegistration.HubConnectionStringKey, ex.Message, StringComparison.Ordinal);
|
||||
Assert.Contains(
|
||||
$"{SecretsRegistration.StoreProviderKey}=SqlServer", ex.Message, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The store's own connection string can never be a <c>${secret:}</c> reference — the pre-host
|
||||
/// expander needs the store this string points at in order to resolve it (bootstrap
|
||||
/// circularity, the same rule as the hub bearer token). Rejected by name rather than left to
|
||||
/// fail as a malformed connection string on first use.
|
||||
/// circularity). Rejected by name rather than left to fail as a malformed connection string on
|
||||
/// first use.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void GrpcMode_Central_WithSecretReferenceConnectionString_FailsAtRegistration()
|
||||
public void Central_StoreProviderSqlServer_SecretReferenceConnectionString_Throws()
|
||||
{
|
||||
var services = new ServiceCollection();
|
||||
services.AddLogging();
|
||||
|
||||
IConfiguration config = BuildConfig(
|
||||
("Secrets:Replication:Enabled", "true"),
|
||||
("Secrets:Replication:Mode", "Grpc"),
|
||||
("Secrets:GrpcHub:BearerToken", DummyBearerToken),
|
||||
("Secrets:Store:Provider", "SqlServer"),
|
||||
("Secrets:SqlServer:ConnectionString", "${secret:hub-connection-string}"));
|
||||
|
||||
var ex = Assert.Throws<InvalidOperationException>(
|
||||
@@ -477,226 +226,120 @@ public class SecretsReplicationWiringTests
|
||||
Assert.DoesNotContain("hub-connection-string", ex.Message, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The sites-don't-talk-to-SQL pin: a site node's container must contain NOTHING from the
|
||||
/// SQL-Server replicator package — not just resolve a SQLite store, but hold no descriptor
|
||||
/// that could ever open a connection to central's database.
|
||||
/// </summary>
|
||||
// ---------------------------------------------------------------------------------------
|
||||
// The Layer-A pre-host ${secret:} expander (SecretsRegistration.AddPreHostExpander, called
|
||||
// from Program.cs) — review round 1, Important #1. It MUST compose against the SAME store
|
||||
// AddScadaBridgeSecrets registers for the running node: an expander that resolved boot-time
|
||||
// ${secret:} tokens (e.g. ScadaBridge:Security:JwtSigningKey) from the per-node SQLite file
|
||||
// while the running node served the shared SQL store would let two central nodes disagree on
|
||||
// a rotated value — exactly the per-node drift Store:Provider=SqlServer exists to prevent.
|
||||
// UsesCentralSharedStore is the single predicate shared by both methods.
|
||||
// ---------------------------------------------------------------------------------------
|
||||
|
||||
[Fact]
|
||||
public void GrpcMode_Site_HasNoSqlServerPackageTypesInTheContainer()
|
||||
public void UsesCentralSharedStore_TrueOnlyForCentralWithProviderSqlServer()
|
||||
{
|
||||
IConfiguration sqlServerConfig = BuildConfig(("Secrets:Store:Provider", "SqlServer"));
|
||||
IConfiguration sqliteConfig = BuildConfig();
|
||||
|
||||
Assert.True(
|
||||
SecretsRegistration.UsesCentralSharedStore(sqlServerConfig, SecretsNodeRole.Central));
|
||||
Assert.False(
|
||||
SecretsRegistration.UsesCentralSharedStore(sqlServerConfig, SecretsNodeRole.Site));
|
||||
Assert.False(
|
||||
SecretsRegistration.UsesCentralSharedStore(sqliteConfig, SecretsNodeRole.Central));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ExpanderCentral_StoreProviderSqlServer_ResolvesSharedSqlStore()
|
||||
{
|
||||
// The bug this guards against: the expander used to be wired ONLY to the retired Grpc
|
||||
// mode, so a Central node on Store:Provider=SqlServer (the retirement message's own
|
||||
// prescribed replacement) resolved boot-time ${secret:} tokens from the wrong store.
|
||||
IConfiguration config = BuildConfig(
|
||||
("Secrets:Store:Provider", "SqlServer"),
|
||||
("Secrets:SqlServer:ConnectionString", DummyConnectionString));
|
||||
|
||||
var services = new ServiceCollection();
|
||||
services.AddLogging();
|
||||
services.AddPreHostExpander(config, SecretsNodeRole.Central, NewContentRoot());
|
||||
|
||||
using ServiceProvider provider = services.BuildServiceProvider(validateScopes: true);
|
||||
|
||||
Assert.IsType<SqlServerSecretStore>(provider.GetRequiredService<ISecretStore>());
|
||||
// The store-agnostic migrator seam — resolving through it is what lets Program.cs's
|
||||
// expander block stay store-agnostic instead of re-deriving which concrete migrator to
|
||||
// ask for.
|
||||
Assert.NotNull(provider.GetRequiredService<ISecretsStoreMigrator>());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ExpanderCentral_Default_ResolvesSqliteStore()
|
||||
{
|
||||
var services = new ServiceCollection();
|
||||
services.AddLogging();
|
||||
services.AddScadaBridgeSecrets(BuildConfig(GrpcSiteConfig()), SecretsNodeRole.Site);
|
||||
|
||||
Assert.DoesNotContain(services, IsFromSqlServerReplicatorPackage);
|
||||
services.AddPreHostExpander(BuildConfig(), SecretsNodeRole.Central, NewContentRoot());
|
||||
|
||||
using ServiceProvider provider = services.BuildServiceProvider(validateScopes: true);
|
||||
|
||||
Assert.IsType<SqliteSecretStore>(provider.GetRequiredService<ISecretStore>());
|
||||
Assert.Null(provider.GetService<SqlServerSecretStore>());
|
||||
Assert.NotNull(provider.GetRequiredService<ISecretsStoreMigrator>());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void GrpcMode_Site_WithoutEndpoint_FailsAtRegistration()
|
||||
public void ExpanderSite_StoreProviderSqlServer_StillResolvesSqliteStore()
|
||||
{
|
||||
// A Site ignores Store:Provider entirely, exactly as the runtime registration does — the
|
||||
// expander must not be a second place where that rule could be got wrong.
|
||||
IConfiguration config = BuildConfig(("Secrets:Store:Provider", "SqlServer"));
|
||||
|
||||
var services = new ServiceCollection();
|
||||
services.AddLogging();
|
||||
services.AddPreHostExpander(config, SecretsNodeRole.Site, NewContentRoot());
|
||||
|
||||
Assert.DoesNotContain(services, IsFromSqlServerPackage);
|
||||
|
||||
using ServiceProvider provider = services.BuildServiceProvider(validateScopes: true);
|
||||
|
||||
Assert.IsType<SqliteSecretStore>(provider.GetRequiredService<ISecretStore>());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ExpanderCentral_StoreProviderSqlServer_BlankConnectionString_Throws()
|
||||
{
|
||||
var services = new ServiceCollection();
|
||||
services.AddLogging();
|
||||
|
||||
IConfiguration config = BuildConfig(
|
||||
("Secrets:Replication:Enabled", "true"),
|
||||
("Secrets:Replication:Mode", "Grpc"),
|
||||
("Secrets:GrpcHub:BearerToken", DummyBearerToken));
|
||||
IConfiguration config = BuildConfig(("Secrets:Store:Provider", "SqlServer"));
|
||||
|
||||
var ex = Assert.Throws<InvalidOperationException>(
|
||||
() => services.AddScadaBridgeSecrets(config, SecretsNodeRole.Site));
|
||||
() => services.AddPreHostExpander(config, SecretsNodeRole.Central, NewContentRoot()));
|
||||
|
||||
Assert.Contains("Secrets:GrpcHub:Endpoint", ex.Message, StringComparison.Ordinal);
|
||||
Assert.Contains(
|
||||
SecretsRegistration.HubConnectionStringKey, ex.Message, StringComparison.Ordinal);
|
||||
Assert.Contains(
|
||||
$"{SecretsRegistration.StoreProviderKey}=SqlServer", ex.Message, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void GrpcMode_Site_WithoutBearerToken_FailsAtRegistration()
|
||||
public void ExpanderCentral_StoreProviderSqlServer_SecretReferenceConnectionString_Throws()
|
||||
{
|
||||
// The bootstrap-circularity rule applies here too: this expander is exactly the code that
|
||||
// would otherwise have to resolve the reference in order to build the store that resolves
|
||||
// it.
|
||||
var services = new ServiceCollection();
|
||||
services.AddLogging();
|
||||
|
||||
IConfiguration config = BuildConfig(
|
||||
("Secrets:Replication:Enabled", "true"),
|
||||
("Secrets:Replication:Mode", "Grpc"),
|
||||
("Secrets:GrpcHub:Endpoint", DummyHubEndpoint));
|
||||
("Secrets:Store:Provider", "SqlServer"),
|
||||
("Secrets:SqlServer:ConnectionString", "${secret:hub-connection-string}"));
|
||||
|
||||
var ex = Assert.Throws<InvalidOperationException>(
|
||||
() => services.AddScadaBridgeSecrets(config, SecretsNodeRole.Site));
|
||||
() => services.AddPreHostExpander(config, SecretsNodeRole.Central, NewContentRoot()));
|
||||
|
||||
Assert.Contains("Secrets:GrpcHub:BearerToken", ex.Message, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------------------
|
||||
// Mode key parsing.
|
||||
// ---------------------------------------------------------------------------------------
|
||||
|
||||
[Theory]
|
||||
[InlineData(null, SecretsReplicationMode.SqlServer)]
|
||||
[InlineData("", SecretsReplicationMode.SqlServer)]
|
||||
[InlineData("SqlServer", SecretsReplicationMode.SqlServer)]
|
||||
[InlineData("sqlserver", SecretsReplicationMode.SqlServer)]
|
||||
[InlineData("Grpc", SecretsReplicationMode.Grpc)]
|
||||
[InlineData("grpc", SecretsReplicationMode.Grpc)]
|
||||
[InlineData(" Grpc ", SecretsReplicationMode.Grpc)]
|
||||
public void ResolveReplicationMode_ReadsTheKey(string? value, SecretsReplicationMode expected)
|
||||
{
|
||||
IConfiguration config = value is null
|
||||
? BuildConfig()
|
||||
: BuildConfig(("Secrets:Replication:Mode", value));
|
||||
|
||||
Assert.Equal(expected, SecretsRegistration.ResolveReplicationMode(config));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ResolveReplicationMode_RejectsAnUnknownValue_EvenWithReplicationOff()
|
||||
{
|
||||
// Caught at the boot that introduces the typo rather than at whichever later boot flips
|
||||
// Secrets:Replication:Enabled — otherwise an unrelated change fails on a key nobody touched.
|
||||
IConfiguration config = BuildConfig(("Secrets:Replication:Mode", "GrpcHub"));
|
||||
|
||||
var ex = Assert.Throws<InvalidOperationException>(
|
||||
() => SecretsRegistration.ResolveReplicationMode(config));
|
||||
|
||||
Assert.Contains("Secrets:Replication:Mode", ex.Message, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A numeric value must not be accepted. <c>Enum.TryParse</c> happily parses any integer,
|
||||
/// including ones outside the enum, so "2" would otherwise select a mode that does not exist
|
||||
/// and fall through the switch to the SQL-Server branch.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void ResolveReplicationMode_RejectsAnOutOfRangeNumericValue()
|
||||
{
|
||||
IConfiguration config = BuildConfig(("Secrets:Replication:Mode", "7"));
|
||||
|
||||
Assert.Throws<InvalidOperationException>(
|
||||
() => SecretsRegistration.ResolveReplicationMode(config));
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Endpoint-mapping pins for the central-hosted secrets hub.
|
||||
/// <para>
|
||||
/// Registration and mapping are gated by one shared predicate on purpose, and this asserts the
|
||||
/// consequence: the hub endpoint exists exactly when the node that serves it registered the
|
||||
/// fail-closed interceptor. A mapped-but-unregistered hub would be an anonymous endpoint serving
|
||||
/// every secret central holds, and it would look completely healthy.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public class SecretsHubEndpointMappingTests
|
||||
{
|
||||
private const string DummyBearerToken = "test-hub-token";
|
||||
|
||||
private static WebApplication BuildCentralApp(params (string Key, string Value)[] overrides)
|
||||
{
|
||||
var builder = WebApplication.CreateBuilder();
|
||||
builder.Configuration.Sources.Clear();
|
||||
|
||||
var settings = new Dictionary<string, string?>
|
||||
{
|
||||
["Secrets:SqlitePath"] = Path.Combine(
|
||||
Path.GetTempPath(), $"sb-secrets-map-test-{Guid.NewGuid():N}.db"),
|
||||
["Secrets:MasterKey:Source"] = "Environment",
|
||||
["Secrets:MasterKey:EnvVarName"] = "ZB_SECRETS_MASTER_KEY",
|
||||
["Secrets:RunMigrationsOnStartup"] = "false",
|
||||
};
|
||||
|
||||
foreach ((string key, string value) in overrides)
|
||||
{
|
||||
settings[key] = value;
|
||||
}
|
||||
|
||||
builder.Configuration.AddInMemoryCollection(settings);
|
||||
builder.Services.AddGrpc();
|
||||
builder.Services.AddScadaBridgeSecrets(builder.Configuration, SecretsNodeRole.Central);
|
||||
|
||||
WebApplication app = builder.Build();
|
||||
app.MapScadaBridgeSecretsHub(app.Configuration);
|
||||
return app;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The gRPC service's WIRE name, from <c>package zb.mom.ww.secrets.hub.v1; service SecretsHub</c>.
|
||||
/// Matched on the route pattern rather than on the C# type, because the route is the thing a
|
||||
/// follower actually addresses and the thing a rename would break.
|
||||
/// </summary>
|
||||
private const string HubRoutePrefix = "/zb.mom.ww.secrets.hub.v1.SecretsHub/";
|
||||
|
||||
private static IReadOnlyList<string> HubRoutes(WebApplication app) =>
|
||||
((IEndpointRouteBuilder)app).DataSources
|
||||
.SelectMany(source => source.Endpoints)
|
||||
.OfType<RouteEndpoint>()
|
||||
.Select(endpoint => "/" + endpoint.RoutePattern.RawText?.TrimStart('/'))
|
||||
.Where(route => route.StartsWith(HubRoutePrefix, StringComparison.Ordinal))
|
||||
// Grpc.AspNetCore always adds a "{unimplementedMethod}" catch-all per service, which
|
||||
// answers Unimplemented. It is framework scaffolding, not an RPC the contract declares.
|
||||
.Where(route => !route.Contains('{', StringComparison.Ordinal))
|
||||
.ToList();
|
||||
|
||||
private static bool HasHubEndpoint(WebApplication app) => HubRoutes(app).Count > 0;
|
||||
|
||||
// Syntactically valid, deliberately unreachable — central + gRPC requires the shared
|
||||
// SQL-Server store's connection string at registration; nothing here ever connects.
|
||||
private const string DummyConnectionString =
|
||||
"Server=unused;Database=x;Integrated Security=true;";
|
||||
|
||||
[Fact]
|
||||
public void GrpcMode_Central_MapsTheHubEndpoint()
|
||||
{
|
||||
using WebApplication app = BuildCentralApp(
|
||||
("Secrets:Replication:Enabled", "true"),
|
||||
("Secrets:Replication:Mode", "Grpc"),
|
||||
("Secrets:GrpcHub:BearerToken", DummyBearerToken),
|
||||
("Secrets:SqlServer:ConnectionString", DummyConnectionString));
|
||||
|
||||
Assert.True(HasHubEndpoint(app));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Pull-only is a property of the wire contract, and this is where it becomes observable to
|
||||
/// ScadaBridge: central maps exactly the two READ methods and nothing that could accept a
|
||||
/// pushed row. If a future package version adds a write RPC, this fails here rather than
|
||||
/// silently opening a path for a site to overwrite central's secrets.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void GrpcMode_Central_MapsOnlyTheTwoReadMethods()
|
||||
{
|
||||
using WebApplication app = BuildCentralApp(
|
||||
("Secrets:Replication:Enabled", "true"),
|
||||
("Secrets:Replication:Mode", "Grpc"),
|
||||
("Secrets:GrpcHub:BearerToken", DummyBearerToken),
|
||||
("Secrets:SqlServer:ConnectionString", DummyConnectionString));
|
||||
|
||||
Assert.Equal(
|
||||
[HubRoutePrefix + "GetManifest", HubRoutePrefix + "GetSecrets"],
|
||||
HubRoutes(app).Order(StringComparer.Ordinal).ToArray());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ReplicationDisabled_MapsNoHubEndpoint()
|
||||
{
|
||||
using WebApplication app = BuildCentralApp(
|
||||
("Secrets:Replication:Enabled", "false"),
|
||||
("Secrets:Replication:Mode", "Grpc"),
|
||||
("Secrets:GrpcHub:BearerToken", DummyBearerToken));
|
||||
|
||||
Assert.False(HasHubEndpoint(app));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SqlServerMode_MapsNoHubEndpoint()
|
||||
{
|
||||
using WebApplication app = BuildCentralApp(
|
||||
("Secrets:Replication:Enabled", "true"),
|
||||
("Secrets:Replication:Mode", "SqlServer"),
|
||||
("Secrets:SqlServer:ConnectionString", "Server=unused;Database=x;Integrated Security=true;"));
|
||||
|
||||
Assert.False(HasHubEndpoint(app));
|
||||
Assert.Contains(
|
||||
SecretsRegistration.HubConnectionStringKey, ex.Message, StringComparison.Ordinal);
|
||||
Assert.Contains("${secret:", ex.Message, StringComparison.Ordinal);
|
||||
Assert.DoesNotContain("hub-connection-string", ex.Message, StringComparison.Ordinal);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -104,6 +104,21 @@ public class SiteLocalDbCdcRegistrationTests : IDisposable
|
||||
Assert.DoesNotContain(triggers, t => t.StartsWith("__localdb_smtp_configurations_", StringComparison.Ordinal));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ReplicatedNode_InstallsNoCaptureTriggersOnTheNodeLocalMirroredSecretsLedger()
|
||||
{
|
||||
// mirrored_secrets records which names THIS node has mirrored into its own secret
|
||||
// store — per-node fact, not site-wide state. Capturing it would replicate one node's
|
||||
// mirroring progress onto the other, so it must stay uncaptured even on a fully
|
||||
// replicated node.
|
||||
var db = BuildDatabase(Config(apiKey: "cdc-test-key"));
|
||||
|
||||
var triggers = CaptureTriggers(db);
|
||||
|
||||
Assert.DoesNotContain(triggers, t => t.StartsWith("__localdb_mirrored_secrets_", StringComparison.Ordinal));
|
||||
Assert.DoesNotContain("mirrored_secrets", db.ReplicatedTables.Keys);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void UnreplicatedNode_DropsTriggersLeftByAPreviouslyReplicatedBuild()
|
||||
{
|
||||
@@ -196,13 +211,18 @@ public class SiteLocalDbCdcRegistrationTests : IDisposable
|
||||
|
||||
// ---- helpers ----------------------------------------------------------------------
|
||||
|
||||
/// <summary>The ten replicated tables plus the two deliberately-unregistered ones.</summary>
|
||||
/// <summary>
|
||||
/// The eleven replicated tables, plus the two deliberately-unregistered central-only
|
||||
/// notification tables, plus the node-local (also deliberately-unregistered)
|
||||
/// <c>mirrored_secrets</c> ledger.
|
||||
/// </summary>
|
||||
private static readonly string[] AllSiteTables =
|
||||
[
|
||||
"OperationTracking", "site_events", "sf_messages", "deployed_configurations",
|
||||
"static_attribute_overrides", "shared_scripts", "external_systems",
|
||||
"database_connections", "data_connection_definitions", "native_alarm_state",
|
||||
"notification_lists", "smtp_configurations",
|
||||
"deployed_secrets",
|
||||
"notification_lists", "smtp_configurations", "mirrored_secrets",
|
||||
];
|
||||
|
||||
private static readonly string[] ReplicatedTables =
|
||||
@@ -210,6 +230,7 @@ public class SiteLocalDbCdcRegistrationTests : IDisposable
|
||||
"OperationTracking", "site_events", "sf_messages", "deployed_configurations",
|
||||
"static_attribute_overrides", "shared_scripts", "external_systems",
|
||||
"database_connections", "data_connection_definitions", "native_alarm_state",
|
||||
"deployed_secrets",
|
||||
];
|
||||
|
||||
private static void AssertCaptureTriggersCoverTheReplicatedTables(ILocalDb db)
|
||||
|
||||
@@ -15,7 +15,8 @@ namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
|
||||
/// <c>RegisterReplicated</c> time, which on a site node is a BOOT failure of the replicating
|
||||
/// pair. This test moves that discovery from the first replicated boot to the build: it walks
|
||||
/// the real site schema (every table <c>SiteLocalDbSetup.OnReady</c> creates) and asserts none
|
||||
/// of the ten <see cref="SiteLocalDbSetup.ReplicatedTables"/> has such an inbound FK.
|
||||
/// of the eleven <see cref="SiteLocalDbSetup.ReplicatedTables"/> (incl. <c>deployed_secrets</c>,
|
||||
/// added for site-secrets artifact deploy) has such an inbound FK.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The schema is built with an UNREPLICATED config on purpose: a replicated one would make
|
||||
|
||||
@@ -245,7 +245,7 @@ public class SiteLocalDbWiringTests : IDisposable
|
||||
|
||||
var tables = TableNames(db);
|
||||
|
||||
// One from each Phase 1 schema, then every Phase 2 table.
|
||||
// One from each Phase 1 schema, then every Phase 2 table plus the site-secrets tables.
|
||||
Assert.Contains("OperationTracking", tables);
|
||||
Assert.Contains("site_events", tables);
|
||||
Assert.Contains("sf_messages", tables);
|
||||
@@ -254,6 +254,7 @@ public class SiteLocalDbWiringTests : IDisposable
|
||||
"deployed_configurations", "static_attribute_overrides", "shared_scripts",
|
||||
"external_systems", "database_connections", "notification_lists",
|
||||
"data_connection_definitions", "smtp_configurations", "native_alarm_state",
|
||||
"deployed_secrets", "mirrored_secrets",
|
||||
})
|
||||
{
|
||||
Assert.Contains(table, tables);
|
||||
@@ -261,19 +262,21 @@ public class SiteLocalDbWiringTests : IDisposable
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Site_LocalDb_RegistersExactlyTheTenReplicatedTables()
|
||||
public void Site_LocalDb_RegistersExactlyTheElevenReplicatedTables()
|
||||
{
|
||||
// The Phase 2 cutover. This is an EXACTLY check rather than a Contains in both
|
||||
// directions, and both directions are load-bearing.
|
||||
// The Phase 2 cutover, extended by site-secrets artifact deploy's deployed_secrets.
|
||||
// This is an EXACTLY check rather than a Contains in both directions, and both
|
||||
// directions are load-bearing.
|
||||
//
|
||||
// Too few: the table silently stops replicating. Storage still works, every other
|
||||
// test still passes, and the pair just quietly diverges — the failure mode Phase 1
|
||||
// existed to end.
|
||||
//
|
||||
// Too many: notification_lists and smtp_configurations must NOT be here. They are
|
||||
// permanently empty by design, and registering them would open a standing
|
||||
// replication channel whose only historical payload was plaintext SMTP passwords.
|
||||
// A Contains-based test would never catch that.
|
||||
// Too many: notification_lists, smtp_configurations, and mirrored_secrets must NOT
|
||||
// be here. notification_lists/smtp_configurations are permanently empty by design
|
||||
// (their only historical payload was a plaintext SMTP password); mirrored_secrets is
|
||||
// node-local fact (which names THIS node has mirrored), not site-wide state. A
|
||||
// Contains-based test would never catch either.
|
||||
var db = _host.Services.GetRequiredService<ILocalDb>();
|
||||
|
||||
Assert.Equal(
|
||||
@@ -281,8 +284,9 @@ public class SiteLocalDbWiringTests : IDisposable
|
||||
// Ordinal order: '_' (0x5F) sorts before 'b' (0x62), so
|
||||
// data_connection_definitions precedes database_connections.
|
||||
"OperationTracking", "data_connection_definitions", "database_connections",
|
||||
"deployed_configurations", "external_systems", "native_alarm_state",
|
||||
"sf_messages", "shared_scripts", "site_events", "static_attribute_overrides",
|
||||
"deployed_configurations", "deployed_secrets", "external_systems",
|
||||
"native_alarm_state", "sf_messages", "shared_scripts", "site_events",
|
||||
"static_attribute_overrides",
|
||||
],
|
||||
db.ReplicatedTables.Keys.OrderBy(k => k, StringComparer.Ordinal).ToArray());
|
||||
}
|
||||
@@ -300,6 +304,17 @@ public class SiteLocalDbWiringTests : IDisposable
|
||||
Assert.DoesNotContain("smtp_configurations", db.ReplicatedTables.Keys);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Site_LocalDb_DoesNotReplicateTheNodeLocalMirroredSecretsLedger()
|
||||
{
|
||||
// mirrored_secrets exists (SiteStorageSchema creates it) but must never be captured:
|
||||
// it records which names THIS node has mirrored into its own secret store, which is
|
||||
// per-node fact, not site-wide state — see SiteLocalDbSetup.ReplicatedTables.
|
||||
var db = _host.Services.GetRequiredService<ILocalDb>();
|
||||
|
||||
Assert.DoesNotContain("mirrored_secrets", db.ReplicatedTables.Keys);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Site_LocalDb_ReplicatedPhase2TablesHaveTheirExpectedPrimaryKeys()
|
||||
{
|
||||
@@ -318,6 +333,7 @@ public class SiteLocalDbWiringTests : IDisposable
|
||||
Assert.Equal(
|
||||
["instance_unique_name", "source_canonical_name", "source_reference"],
|
||||
db.ReplicatedTables["native_alarm_state"].PkColumns);
|
||||
Assert.Equal(["name"], db.ReplicatedTables["deployed_secrets"].PkColumns);
|
||||
}
|
||||
|
||||
private static HashSet<string> TableNames(ILocalDb db)
|
||||
|
||||
@@ -699,7 +699,11 @@ public class StartupValidatorTests
|
||||
{
|
||||
// Design 2026-09-02: the inbound API key store is per-node SQLite by default, so a
|
||||
// 2+-node central cluster is one failover away from rejecting every inbound client.
|
||||
var config = BuildConfig(ValidCentralConfig());
|
||||
// Secrets:Store:Provider is set to SqlServer here to isolate this warning from the
|
||||
// independent one added for task 5 (ClusteredCentral_SqliteSecretStore_Warns below).
|
||||
var values = ValidCentralConfig();
|
||||
values["Secrets:Store:Provider"] = "SqlServer";
|
||||
var config = BuildConfig(values);
|
||||
|
||||
var warnings = StartupValidator.CollectWarnings(config);
|
||||
|
||||
@@ -714,6 +718,7 @@ public class StartupValidatorTests
|
||||
{
|
||||
var values = ValidCentralConfig();
|
||||
values["ScadaBridge:InboundApi:ApiKeyStore:Provider"] = "sqlite";
|
||||
values["Secrets:Store:Provider"] = "SqlServer";
|
||||
var config = BuildConfig(values);
|
||||
|
||||
var warnings = StartupValidator.CollectWarnings(config);
|
||||
@@ -726,6 +731,7 @@ public class StartupValidatorTests
|
||||
{
|
||||
var values = ValidCentralConfig();
|
||||
values["ScadaBridge:InboundApi:ApiKeyStore:Provider"] = "SqlServer";
|
||||
values["Secrets:Store:Provider"] = "SqlServer";
|
||||
var config = BuildConfig(values);
|
||||
|
||||
var warnings = StartupValidator.CollectWarnings(config);
|
||||
@@ -733,6 +739,208 @@ public class StartupValidatorTests
|
||||
Assert.Empty(warnings);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ClusteredCentral_SqliteSecretStore_Warns()
|
||||
{
|
||||
// Task 5: a clustered central left on the per-node SQLite secret store silently drifts
|
||||
// the moment anyone edits a secret on only one node, then ships an inconsistent set to
|
||||
// sites. ApiKeyStore:Provider is set to SqlServer here to isolate this warning from the
|
||||
// pre-existing one above.
|
||||
var values = ValidCentralConfig();
|
||||
values["ScadaBridge:InboundApi:ApiKeyStore:Provider"] = "SqlServer";
|
||||
var config = BuildConfig(values);
|
||||
|
||||
var warnings = StartupValidator.CollectWarnings(config);
|
||||
|
||||
var warning = Assert.Single(warnings);
|
||||
Assert.Contains("clustered central", warning);
|
||||
Assert.Contains("SQLite", warning);
|
||||
Assert.Contains("Secrets:Store:Provider=SqlServer", warning);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ClusteredCentral_SqlServerSecretStore_Silent()
|
||||
{
|
||||
var values = ValidCentralConfig();
|
||||
values["ScadaBridge:InboundApi:ApiKeyStore:Provider"] = "SqlServer";
|
||||
values["Secrets:Store:Provider"] = "SqlServer";
|
||||
var config = BuildConfig(values);
|
||||
|
||||
var warnings = StartupValidator.CollectWarnings(config);
|
||||
|
||||
Assert.Empty(warnings);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Task 11: secrets replication (the gRPC hub, and the SQL-Server ReplicatingSecretStore mode
|
||||
/// that preceded it) is retired in ScadaBridge — sites receive secrets exclusively via Deploy
|
||||
/// Artifacts, and a replicated site store would publish DeployedSecretsMirror's local deletes
|
||||
/// fleet-wide. Checked on BOTH roles: a Site used to be the gRPC hub's follower half, so the
|
||||
/// flag was never Central-only. Replaces the old warning-suppression test
|
||||
/// (ClusteredCentral_ReplicationEnabled_NoSecretsStoreWarning) — that suppression is gone too,
|
||||
/// because CollectWarnings never runs with this flag true on a config that also passed
|
||||
/// Validate.
|
||||
/// </summary>
|
||||
[Theory]
|
||||
[InlineData("true")]
|
||||
[InlineData("True")]
|
||||
// Review round 1, Minor #3: these are not valid booleans. GetValue<bool> used to throw a raw
|
||||
// OptionsValidationException-style binder crash on them instead of the designed message; the
|
||||
// key is read as a raw string now, so any non-"false" value is "set" and rejected the same way.
|
||||
[InlineData("yes")]
|
||||
[InlineData("1")]
|
||||
public void ReplicationEnabled_IsRejected_OnCentral(string value)
|
||||
{
|
||||
var values = ValidCentralConfig();
|
||||
values["Secrets:Replication:Enabled"] = value;
|
||||
var config = BuildConfig(values);
|
||||
|
||||
var ex = Assert.Throws<InvalidOperationException>(() => StartupValidator.Validate(config));
|
||||
|
||||
Assert.Contains(
|
||||
"Secrets replication (the gRPC hub) is retired in ScadaBridge", ex.Message);
|
||||
Assert.Contains("sites receive secrets with Deploy Artifacts", ex.Message);
|
||||
Assert.Contains("central uses Secrets:Store:Provider=SqlServer", ex.Message);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ReplicationEnabled_IsRejected_OnSite()
|
||||
{
|
||||
var values = ValidSiteConfig();
|
||||
values["Secrets:Replication:Enabled"] = "true";
|
||||
var config = BuildConfig(values);
|
||||
|
||||
var ex = Assert.Throws<InvalidOperationException>(() => StartupValidator.Validate(config));
|
||||
|
||||
Assert.Contains(
|
||||
"Secrets replication (the gRPC hub) is retired in ScadaBridge", ex.Message);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ReplicationDisabled_PassesValidation()
|
||||
{
|
||||
// Explicit false must not be confused with "set" — the default posture every existing
|
||||
// deployment ships stays valid.
|
||||
var values = ValidCentralConfig();
|
||||
values["Secrets:Replication:Enabled"] = "false";
|
||||
var config = BuildConfig(values);
|
||||
|
||||
var ex = Record.Exception(() => StartupValidator.Validate(config));
|
||||
|
||||
Assert.Null(ex);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ReplicationDisabled_CaseInsensitive_PassesValidation()
|
||||
{
|
||||
var values = ValidCentralConfig();
|
||||
values["Secrets:Replication:Enabled"] = "FALSE";
|
||||
var config = BuildConfig(values);
|
||||
|
||||
var ex = Record.Exception(() => StartupValidator.Validate(config));
|
||||
|
||||
Assert.Null(ex);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Deliberately stricter than <c>bool</c> parsing (review round 1, Minor #3): only an absent
|
||||
/// key or exactly (case-insensitively) <c>"false"</c> is safe. <c>"0"</c> is a legal
|
||||
/// <c>bool.TryParse</c>-adjacent falsy spelling in some conventions but is NOT exempted here —
|
||||
/// a leftover key this malformed should surface the retirement message, not silently pass.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void ReplicationEnabled_Zero_IsRejected()
|
||||
{
|
||||
var values = ValidCentralConfig();
|
||||
values["Secrets:Replication:Enabled"] = "0";
|
||||
var config = BuildConfig(values);
|
||||
|
||||
var ex = Assert.Throws<InvalidOperationException>(() => StartupValidator.Validate(config));
|
||||
|
||||
Assert.Contains(
|
||||
"Secrets replication (the gRPC hub) is retired in ScadaBridge", ex.Message);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Review round 1, Minor #4: a leftover <c>Secrets:Replication</c>/<c>Secrets:GrpcHub</c> key
|
||||
/// from before Task 11 is not itself a boot failure (Validate already rejects Enabled=true
|
||||
/// outright), but every OTHER key under either section is now silently ignored — worth a
|
||||
/// Warning so an operator cleans up QA/prod appsettings.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void CollectWarnings_WarnsForLeftoverReplicationModeKey()
|
||||
{
|
||||
var values = ValidCentralConfig();
|
||||
values["Secrets:Store:Provider"] = "SqlServer";
|
||||
values["Secrets:Replication:Mode"] = "SqlServer";
|
||||
var config = BuildConfig(values);
|
||||
|
||||
var warnings = StartupValidator.CollectWarnings(config);
|
||||
|
||||
Assert.Contains(warnings, w =>
|
||||
w.Contains("Secrets:Replication / Secrets:GrpcHub settings are ignored", StringComparison.Ordinal)
|
||||
&& w.Contains("the secrets hub is retired; remove them", StringComparison.Ordinal));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void CollectWarnings_WarnsForLeftoverGrpcHubKey()
|
||||
{
|
||||
var values = ValidCentralConfig();
|
||||
values["Secrets:Store:Provider"] = "SqlServer";
|
||||
values["Secrets:GrpcHub:BearerToken"] = "stale-leftover-token";
|
||||
var config = BuildConfig(values);
|
||||
|
||||
var warnings = StartupValidator.CollectWarnings(config);
|
||||
|
||||
Assert.Contains(warnings, w =>
|
||||
w.Contains("Secrets:Replication / Secrets:GrpcHub settings are ignored", StringComparison.Ordinal));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void CollectWarnings_WarnsForLeftoverKeys_OnSiteToo()
|
||||
{
|
||||
// The warning is not central-scoped: a Site config can carry leftover GrpcHub follower
|
||||
// settings just as easily.
|
||||
var values = ValidSiteConfig();
|
||||
values["Secrets:GrpcHub:Endpoint"] = "http://stale-central:8083";
|
||||
var config = BuildConfig(values);
|
||||
|
||||
var warnings = StartupValidator.CollectWarnings(config);
|
||||
|
||||
Assert.Contains(warnings, w =>
|
||||
w.Contains("Secrets:Replication / Secrets:GrpcHub settings are ignored", StringComparison.Ordinal));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void CollectWarnings_SilentForBareReplicationEnabledFalse()
|
||||
{
|
||||
// "Enabled: false" alone is the normal, still-valid shape of every existing deployment's
|
||||
// config (it is what ships in appsettings.json) — this must NOT warn just because the
|
||||
// Secrets:Replication section exists.
|
||||
var values = ValidCentralConfig();
|
||||
values["Secrets:Store:Provider"] = "SqlServer";
|
||||
values["Secrets:Replication:Enabled"] = "false";
|
||||
var config = BuildConfig(values);
|
||||
|
||||
var warnings = StartupValidator.CollectWarnings(config);
|
||||
|
||||
Assert.DoesNotContain(warnings, w =>
|
||||
w.Contains("Secrets:Replication / Secrets:GrpcHub settings are ignored", StringComparison.Ordinal));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void CollectWarnings_SilentWhenNeitherSectionPresent()
|
||||
{
|
||||
var values = ValidCentralConfig();
|
||||
values["Secrets:Store:Provider"] = "SqlServer";
|
||||
var config = BuildConfig(values);
|
||||
|
||||
var warnings = StartupValidator.CollectWarnings(config);
|
||||
|
||||
Assert.DoesNotContain(warnings, w =>
|
||||
w.Contains("Secrets:Replication / Secrets:GrpcHub settings are ignored", StringComparison.Ordinal));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void CollectWarnings_SilentForSingleSeedCentral()
|
||||
{
|
||||
@@ -779,6 +987,28 @@ public class StartupValidatorTests
|
||||
Assert.Null(ex);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void UnknownSecretStoreProvider_Fails()
|
||||
{
|
||||
var values = ValidCentralConfig();
|
||||
values["Secrets:Store:Provider"] = "Postgres";
|
||||
var config = BuildConfig(values);
|
||||
|
||||
var ex = Assert.Throws<InvalidOperationException>(() => StartupValidator.Validate(config));
|
||||
Assert.Contains("Secrets:Store:Provider", ex.Message);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Validate_AcceptsSecretStoreProviderCaseInsensitively()
|
||||
{
|
||||
var values = ValidCentralConfig();
|
||||
values["Secrets:Store:Provider"] = "sqlserver";
|
||||
var config = BuildConfig(values);
|
||||
|
||||
var ex = Record.Exception(() => StartupValidator.Validate(config));
|
||||
Assert.Null(ex);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void MultipleErrors_AllReported()
|
||||
{
|
||||
|
||||
@@ -85,7 +85,8 @@ public class ManagementActorTests : TestKit, IDisposable
|
||||
Substitute.For<IAuditService>(),
|
||||
Options.Create(new DeploymentManagerOptions()),
|
||||
NullLogger<ArtifactDeploymentService>.Instance,
|
||||
new ArtifactRevisionHashService());
|
||||
new ArtifactRevisionHashService(),
|
||||
new SiteSecretsCollector(Substitute.For<ZB.MOM.WW.Secrets.Abstractions.ISecretStore>()));
|
||||
_services.AddScoped(_ => _artifactService);
|
||||
}
|
||||
|
||||
|
||||
+385
@@ -0,0 +1,385 @@
|
||||
using Akka.Actor;
|
||||
using Akka.TestKit;
|
||||
using Akka.TestKit.Xunit2;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Secrets;
|
||||
using ZB.MOM.WW.ScadaBridge.TestSupport;
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors;
|
||||
|
||||
/// <summary>
|
||||
/// Task 10, site-secrets artifact deploy: <see cref="DeploymentManagerActor.HandleDeployArtifacts"/>
|
||||
/// writes shipped secrets to <c>deployed_secrets</c>, runs <see cref="DeployedSecretsMirror.ReconcileAsync"/>
|
||||
/// synchronously, and answers with <see cref="ArtifactDeploymentResponse.SecretsApplied"/> or a failure.
|
||||
/// Follows the shape of the existing artifact-apply tests in <c>DeploymentManagerActorTests</c>.
|
||||
/// </summary>
|
||||
public class DeploymentManagerActorArtifactSecretsTests : TestKit, IDisposable
|
||||
{
|
||||
private readonly SiteStorageService _storage;
|
||||
private readonly ScriptCompilationService _compilationService;
|
||||
private readonly SharedScriptLibrary _sharedScriptLibrary;
|
||||
private readonly TestLocalDb _localDb;
|
||||
|
||||
public DeploymentManagerActorArtifactSecretsTests()
|
||||
{
|
||||
_localDb = TestLocalDb.CreateTemp("dm-artifact-secrets-test");
|
||||
_storage = new SiteStorageService(
|
||||
_localDb.Db,
|
||||
NullLogger<SiteStorageService>.Instance);
|
||||
_storage.InitializeAsync().GetAwaiter().GetResult();
|
||||
_compilationService = new ScriptCompilationService(
|
||||
NullLogger<ScriptCompilationService>.Instance);
|
||||
_sharedScriptLibrary = new SharedScriptLibrary(
|
||||
_compilationService, NullLogger<SharedScriptLibrary>.Instance);
|
||||
}
|
||||
|
||||
void IDisposable.Dispose()
|
||||
{
|
||||
Shutdown();
|
||||
var path = _localDb.Path;
|
||||
_localDb.Dispose();
|
||||
TestLocalDb.DeleteFiles(path);
|
||||
}
|
||||
|
||||
private IActorRef CreateDeploymentManager(DeployedSecretsMirror? secretsMirror = null, IActorRef? dclManager = null)
|
||||
{
|
||||
var options = new SiteRuntimeOptions();
|
||||
// Positional, not named, past the constructor's first six parameters: a named
|
||||
// argument for a trailing optional parameter (secretsMirror) that SKIPS other,
|
||||
// still-optional parameters in between is rejected by the compiler inside an
|
||||
// expression-tree lambda (Props.Create's Expression<Func<T>> overload) with
|
||||
// CS9307 "named argument specification out of position".
|
||||
return ActorOf(Props.Create(() => new DeploymentManagerActor(
|
||||
_storage,
|
||||
_compilationService,
|
||||
_sharedScriptLibrary,
|
||||
null, // streamManager: no stream manager in tests
|
||||
options,
|
||||
NullLogger<DeploymentManagerActor>.Instance,
|
||||
dclManager,
|
||||
null, // healthCollector
|
||||
null, // serviceProvider
|
||||
null, // loggerFactory
|
||||
null, // configFetcher
|
||||
null, // startupLoadRetryInterval
|
||||
null, // configLoader
|
||||
null, // externalSystemCache
|
||||
null, // scriptScheduler
|
||||
null, // connectionDefinitionLoader
|
||||
null, // deployedConfigLoader
|
||||
secretsMirror)));
|
||||
}
|
||||
|
||||
private DeployedSecretsMirror CreateMirror(FakeSecretStore store, FakeSecretResolver resolver)
|
||||
{
|
||||
var options = Options.Create(new SiteRuntimeOptions
|
||||
{
|
||||
DeployedSecretsMirrorInterval = TimeSpan.FromSeconds(15),
|
||||
});
|
||||
return new DeployedSecretsMirror(
|
||||
_storage,
|
||||
store,
|
||||
resolver,
|
||||
invalidator: null,
|
||||
options,
|
||||
NullLogger<DeployedSecretsMirror>.Instance,
|
||||
TimeProvider.System);
|
||||
}
|
||||
|
||||
private static DeployArtifactsCommand ArtifactsCommand(
|
||||
string deploymentId, IReadOnlyList<SecretArtifact>? secrets) => new(
|
||||
DeploymentId: deploymentId,
|
||||
SharedScripts: null,
|
||||
ExternalSystems: null,
|
||||
DatabaseConnections: null,
|
||||
NotificationLists: null,
|
||||
DataConnections: null,
|
||||
SmtpConfigurations: null,
|
||||
Timestamp: DateTimeOffset.UtcNow,
|
||||
Secrets: secrets);
|
||||
|
||||
private static SecretArtifact Art(string name, long revision, string kek = "kek-1") => new(
|
||||
Name: name,
|
||||
ContentType: "Text",
|
||||
Ciphertext: [1, 2, 3, 4, 5],
|
||||
Nonce: [10, 11, 12],
|
||||
Tag: [20, 21, 22, 23],
|
||||
WrappedDek: [30, 31, 32, 33, 34, 35],
|
||||
WrapNonce: [40, 41, 42],
|
||||
WrapTag: [50, 51, 52, 53],
|
||||
KekId: kek,
|
||||
Revision: revision,
|
||||
CreatedUtc: new DateTimeOffset(2026, 9, 28, 0, 0, 0, TimeSpan.Zero),
|
||||
UpdatedUtc: new DateTimeOffset(2026, 9, 28, 1, 0, 0, TimeSpan.Zero),
|
||||
Description: "test secret");
|
||||
|
||||
[Fact]
|
||||
public async Task Secrets_Null_LeavesTableUntouched_AndSecretsAppliedNull()
|
||||
{
|
||||
var store = new FakeSecretStore();
|
||||
var resolver = new FakeSecretResolver();
|
||||
var mirror = CreateMirror(store, resolver);
|
||||
var manager = CreateDeploymentManager(mirror);
|
||||
|
||||
manager.Tell(ArtifactsCommand("dep-null", secrets: null));
|
||||
|
||||
var response = ExpectMsg<ArtifactDeploymentResponse>(TimeSpan.FromSeconds(10));
|
||||
Assert.True(response.Success);
|
||||
Assert.Null(response.SecretsApplied);
|
||||
|
||||
var rows = await _storage.GetDeployedSecretsAsync();
|
||||
Assert.Empty(rows);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Secrets_Present_FullSetWrite_SyncReconcile_SecretsAppliedEqualsShippedCount()
|
||||
{
|
||||
var store = new FakeSecretStore();
|
||||
var resolver = new FakeSecretResolver();
|
||||
var mirror = CreateMirror(store, resolver);
|
||||
var manager = CreateDeploymentManager(mirror);
|
||||
|
||||
manager.Tell(ArtifactsCommand("dep-2secrets", secrets: [Art("a", 1), Art("b", 1)]));
|
||||
|
||||
var response = ExpectMsg<ArtifactDeploymentResponse>(TimeSpan.FromSeconds(10));
|
||||
Assert.True(response.Success);
|
||||
Assert.Equal(2, response.SecretsApplied);
|
||||
Assert.Null(response.ErrorMessage);
|
||||
|
||||
// Reconcile ran synchronously before the reply — both secrets already applied.
|
||||
Assert.True(store.Contains("a"));
|
||||
Assert.True(store.Contains("b"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Secrets_Empty_WithMirror_SecretsAppliedZero_Success()
|
||||
{
|
||||
var store = new FakeSecretStore();
|
||||
var resolver = new FakeSecretResolver();
|
||||
var mirror = CreateMirror(store, resolver);
|
||||
var manager = CreateDeploymentManager(mirror);
|
||||
|
||||
manager.Tell(ArtifactsCommand("dep-empty", secrets: []));
|
||||
|
||||
var response = ExpectMsg<ArtifactDeploymentResponse>(TimeSpan.FromSeconds(10));
|
||||
Assert.True(response.Success);
|
||||
Assert.Equal(0, response.SecretsApplied);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Secrets_MirrorFailsOne_SuccessFalse_KekIdInMessage_SecretsAppliedExcludesFailure()
|
||||
{
|
||||
var store = new FakeSecretStore();
|
||||
var resolver = new FakeSecretResolver();
|
||||
resolver.ThrowFor("bad");
|
||||
var mirror = CreateMirror(store, resolver);
|
||||
var manager = CreateDeploymentManager(mirror);
|
||||
|
||||
manager.Tell(ArtifactsCommand(
|
||||
"dep-onefails", secrets: [Art("good", 1), Art("bad", 1, kek: "kek-central")]));
|
||||
|
||||
var response = ExpectMsg<ArtifactDeploymentResponse>(TimeSpan.FromSeconds(10));
|
||||
Assert.False(response.Success);
|
||||
Assert.NotNull(response.ErrorMessage);
|
||||
Assert.Contains("kek-central", response.ErrorMessage);
|
||||
Assert.Contains("'bad'", response.ErrorMessage);
|
||||
Assert.Equal(1, response.SecretsApplied); // "good" applied+verified; "bad" failed
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Secrets_NonEmpty_NoMirrorRegistered_FailsDeploy()
|
||||
{
|
||||
var manager = CreateDeploymentManager(secretsMirror: null);
|
||||
|
||||
manager.Tell(ArtifactsCommand("dep-nomirror", secrets: [Art("a", 1)]));
|
||||
|
||||
var response = ExpectMsg<ArtifactDeploymentResponse>(TimeSpan.FromSeconds(10));
|
||||
Assert.False(response.Success);
|
||||
Assert.Equal("Secret deployment is not enabled on this site node", response.ErrorMessage);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Secrets_Empty_NoMirrorRegistered_TreatedAsZeroSuccess()
|
||||
{
|
||||
var manager = CreateDeploymentManager(secretsMirror: null);
|
||||
|
||||
manager.Tell(ArtifactsCommand("dep-nomirror-empty", secrets: []));
|
||||
|
||||
var response = ExpectMsg<ArtifactDeploymentResponse>(TimeSpan.FromSeconds(10));
|
||||
Assert.True(response.Success);
|
||||
Assert.Equal(0, response.SecretsApplied);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Secrets_RemovalFailure_DoesNotFailDeploy_AndExcludedFromErrorMessage()
|
||||
{
|
||||
// A row that was previously mirrored (ledgered) but is now tombstoned upstream, and
|
||||
// whose removal fails on this node's store — a failure UNRELATED to the names shipped
|
||||
// by THIS deploy. Per ruling: log at Warning, never surface in ErrorMessage, never fail
|
||||
// the deploy, never affect SecretsApplied (which only counts SHIPPED names).
|
||||
var store = new FakeSecretStore();
|
||||
var resolver = new FakeSecretResolver();
|
||||
var mirror = CreateMirror(store, resolver);
|
||||
var manager = CreateDeploymentManager(mirror);
|
||||
|
||||
// First deploy ships + ledgers "gone".
|
||||
manager.Tell(ArtifactsCommand("dep-r1", secrets: [Art("gone", 1)]));
|
||||
var r1 = ExpectMsg<ArtifactDeploymentResponse>(TimeSpan.FromSeconds(10));
|
||||
Assert.True(r1.Success);
|
||||
Assert.True(store.Contains("gone"));
|
||||
|
||||
// Make the store's delete fail on the next pass, then ship a deploy that no longer
|
||||
// references "gone" (and ships an unrelated, healthy secret).
|
||||
store.ThrowOnDelete = true;
|
||||
manager.Tell(ArtifactsCommand("dep-r2", secrets: [Art("kept", 1)]));
|
||||
var r2 = ExpectMsg<ArtifactDeploymentResponse>(TimeSpan.FromSeconds(10));
|
||||
|
||||
Assert.True(r2.Success);
|
||||
Assert.Equal(1, r2.SecretsApplied); // only "kept" is shipped by this deploy
|
||||
Assert.Null(r2.ErrorMessage);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void DataConnections_PushedToDcl_OnlyAfterShippedSecretsAreMirrored()
|
||||
{
|
||||
// M2 (final review): a connection whose config references a secret shipped by THIS
|
||||
// deploy must find it on its first connect, so the DCL create must not reach the
|
||||
// DCL manager before the secret is in this node's store.
|
||||
var store = new FakeSecretStore();
|
||||
var resolver = new FakeSecretResolver();
|
||||
var mirror = CreateMirror(store, resolver);
|
||||
var dcl = CreateTestProbe();
|
||||
bool? secretPresentWhenDclSawCreate = null;
|
||||
dcl.SetAutoPilot(new DelegateAutoPilot((_, msg) =>
|
||||
{
|
||||
if (msg is ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection.CreateConnectionCommand)
|
||||
{
|
||||
secretPresentWhenDclSawCreate ??= store.Contains("gw-key");
|
||||
}
|
||||
|
||||
return AutoPilot.KeepRunning;
|
||||
}));
|
||||
var manager = CreateDeploymentManager(mirror, dcl.Ref);
|
||||
|
||||
var command = ArtifactsCommand("dep-order", secrets: [Art("gw-key", 1)]) with
|
||||
{
|
||||
DataConnections = [new DataConnectionArtifact("Gw", "OpcUa", "{}", null, 3)],
|
||||
};
|
||||
manager.Tell(command);
|
||||
|
||||
var response = ExpectMsg<ArtifactDeploymentResponse>(TimeSpan.FromSeconds(10));
|
||||
Assert.True(response.Success);
|
||||
Assert.Equal(1, response.SecretsApplied);
|
||||
AwaitCondition(() => secretPresentWhenDclSawCreate.HasValue, TimeSpan.FromSeconds(10));
|
||||
Assert.True(secretPresentWhenDclSawCreate);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task DataConnections_StillStoredAndPushed_WhenSecretsStepFails()
|
||||
{
|
||||
// The reorder must not lose the connection push when the secrets step fails.
|
||||
var dcl = CreateTestProbe();
|
||||
var manager = CreateDeploymentManager(secretsMirror: null, dclManager: dcl.Ref);
|
||||
|
||||
var command = ArtifactsCommand("dep-order-fail", secrets: [Art("gw-key", 1)]) with
|
||||
{
|
||||
DataConnections = [new DataConnectionArtifact("Gw", "OpcUa", "{}", null, 3)],
|
||||
};
|
||||
manager.Tell(command);
|
||||
|
||||
var response = ExpectMsg<ArtifactDeploymentResponse>(TimeSpan.FromSeconds(10));
|
||||
Assert.False(response.Success);
|
||||
dcl.ExpectMsg<ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection.CreateConnectionCommand>(
|
||||
TimeSpan.FromSeconds(10));
|
||||
Assert.Contains(await _storage.GetAllDataConnectionDefinitionsAsync(), d => d.Name == "Gw");
|
||||
}
|
||||
|
||||
// ── Fakes (mirrors DeployedSecretsMirrorTests' fakes; kept local/minimal per-task) ──
|
||||
|
||||
private sealed class FakeSecretStore : ISecretStore
|
||||
{
|
||||
private readonly Dictionary<string, StoredSecret> _rows = new(StringComparer.Ordinal);
|
||||
|
||||
public bool ThrowOnDelete { get; set; }
|
||||
|
||||
public bool Contains(string name) => _rows.TryGetValue(name, out var row) && !row.IsDeleted;
|
||||
|
||||
public Task<StoredSecret?> GetAsync(SecretName name, CancellationToken ct) =>
|
||||
Task.FromResult(_rows.TryGetValue(name.Value, out var row) ? row : null);
|
||||
|
||||
public Task UpsertAsync(StoredSecret row, CancellationToken ct)
|
||||
{
|
||||
_rows[row.Name.Value] = row;
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
public Task<bool> DeleteAsync(SecretName name, string? actor, CancellationToken ct)
|
||||
{
|
||||
if (ThrowOnDelete)
|
||||
{
|
||||
throw new InvalidOperationException("simulated delete fault");
|
||||
}
|
||||
|
||||
if (!_rows.TryGetValue(name.Value, out var existing) || existing.IsDeleted)
|
||||
{
|
||||
return Task.FromResult(false);
|
||||
}
|
||||
|
||||
_rows[name.Value] = existing with
|
||||
{
|
||||
IsDeleted = true,
|
||||
DeletedUtc = DateTimeOffset.UtcNow,
|
||||
UpdatedUtc = DateTimeOffset.UtcNow,
|
||||
Revision = existing.Revision + 1,
|
||||
UpdatedBy = actor,
|
||||
};
|
||||
return Task.FromResult(true);
|
||||
}
|
||||
|
||||
public Task<IReadOnlyList<SecretMetadata>> ListAsync(bool includeDeleted, CancellationToken ct) =>
|
||||
Task.FromResult<IReadOnlyList<SecretMetadata>>([]);
|
||||
|
||||
public Task<IReadOnlyList<SecretManifestEntry>> GetManifestAsync(CancellationToken ct) =>
|
||||
Task.FromResult<IReadOnlyList<SecretManifestEntry>>([]);
|
||||
|
||||
public Task ApplyReplicatedAsync(StoredSecret row, CancellationToken ct)
|
||||
{
|
||||
if (_rows.TryGetValue(row.Name.Value, out var existing)
|
||||
&& !SecretLastWriterWins.IsNewer(row.UpdatedUtc, row.Revision, existing.UpdatedUtc, existing.Revision))
|
||||
{
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
_rows[row.Name.Value] = row;
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
public Task<bool> ApplyRewrapAsync(
|
||||
StoredSecret rewrappedRow, byte[] expectedCurrentWrappedDek, CancellationToken ct) =>
|
||||
Task.FromResult(false);
|
||||
}
|
||||
|
||||
private sealed class FakeSecretResolver : ISecretResolver
|
||||
{
|
||||
private readonly HashSet<string> _throwing = new(StringComparer.Ordinal);
|
||||
|
||||
public void ThrowFor(string name) => _throwing.Add(name);
|
||||
|
||||
public Task<string?> GetAsync(SecretName name, CancellationToken ct)
|
||||
{
|
||||
if (_throwing.Contains(name.Value))
|
||||
{
|
||||
throw new InvalidOperationException("simulated decrypt failure");
|
||||
}
|
||||
|
||||
return Task.FromResult<string?>("decrypted-value");
|
||||
}
|
||||
}
|
||||
}
|
||||
+320
@@ -0,0 +1,320 @@
|
||||
using System.Text.Json;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
|
||||
using ZB.MOM.WW.ScadaBridge.TestSupport;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Persistence;
|
||||
|
||||
/// <summary>
|
||||
/// Task 8, site-secrets artifact deploy: site-local <c>deployed_secrets</c>
|
||||
/// (LocalDb-replicated ciphertext) and <c>mirrored_secrets</c> (node-local ledger) stores.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Backed by a real temp-file LocalDb: <see cref="SiteStorageService"/> takes an
|
||||
/// <c>ILocalDb</c> rather than a connection string, and LocalDb has no in-memory mode.
|
||||
/// </remarks>
|
||||
public class DeployedSecretsStorageTests : IAsyncLifetime, IDisposable
|
||||
{
|
||||
private readonly TestLocalDb _localDb;
|
||||
private SiteStorageService _storage = null!;
|
||||
|
||||
public DeployedSecretsStorageTests()
|
||||
{
|
||||
_localDb = TestLocalDb.CreateTemp("deployed-secrets");
|
||||
}
|
||||
|
||||
public async Task InitializeAsync()
|
||||
{
|
||||
_storage = new SiteStorageService(_localDb.Db, NullLogger<SiteStorageService>.Instance);
|
||||
await _storage.InitializeAsync();
|
||||
}
|
||||
|
||||
public Task DisposeAsync() => Task.CompletedTask;
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
var path = _localDb.Path;
|
||||
_localDb.Dispose();
|
||||
TestLocalDb.DeleteFiles(path);
|
||||
}
|
||||
|
||||
// ── deployed_secrets: full-set upsert / soft-delete reconciliation ──
|
||||
|
||||
[Fact]
|
||||
public async Task Upsert_IsFullSet_MarksAbsentRowsRemoved()
|
||||
{
|
||||
var s = _storage;
|
||||
await s.UpsertDeployedSecretsAsync([Art("A", 1), Art("B", 1)], "d1");
|
||||
var (applied, removed) = await s.UpsertDeployedSecretsAsync([Art("A", 2)], "d2");
|
||||
|
||||
Assert.Equal(["A"], applied);
|
||||
Assert.Equal(["B"], removed);
|
||||
|
||||
var rows = await s.GetDeployedSecretsAsync();
|
||||
Assert.Equal(2, rows.Single(r => r.Name == "A").Secret.Revision);
|
||||
Assert.True(rows.Single(r => r.Name == "B").IsRemoved);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Upsert_FirstCall_AppliesEveryRow_AndRemovesNothing()
|
||||
{
|
||||
var (applied, removed) = await _storage.UpsertDeployedSecretsAsync(
|
||||
[Art("A", 1), Art("B", 1)], "d1");
|
||||
|
||||
Assert.Equal(["A", "B"], applied);
|
||||
Assert.Empty(removed);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Upsert_ReAppliedRow_IsNotReportedAsRemovedTwice()
|
||||
{
|
||||
await _storage.UpsertDeployedSecretsAsync([Art("A", 1), Art("B", 1)], "d1");
|
||||
var (_, removedFirst) = await _storage.UpsertDeployedSecretsAsync([Art("A", 2)], "d2");
|
||||
Assert.Equal(["B"], removedFirst);
|
||||
|
||||
// B stays removed and is not re-reported as "newly" removed on a later call that
|
||||
// still omits it.
|
||||
var (_, removedSecond) = await _storage.UpsertDeployedSecretsAsync([Art("A", 3)], "d3");
|
||||
Assert.Empty(removedSecond);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Upsert_RowThatWasRemoved_CanBeReAppliedAndFlipsBackToNotRemoved()
|
||||
{
|
||||
await _storage.UpsertDeployedSecretsAsync([Art("A", 1), Art("B", 1)], "d1");
|
||||
await _storage.UpsertDeployedSecretsAsync([Art("A", 1)], "d2"); // B removed
|
||||
|
||||
var (applied, removed) = await _storage.UpsertDeployedSecretsAsync(
|
||||
[Art("A", 1), Art("B", 2)], "d3");
|
||||
|
||||
Assert.Equal(["A", "B"], applied);
|
||||
Assert.Empty(removed);
|
||||
|
||||
var rows = await _storage.GetDeployedSecretsAsync();
|
||||
var b = rows.Single(r => r.Name == "B");
|
||||
Assert.False(b.IsRemoved);
|
||||
Assert.Equal(2, b.Secret.Revision);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Upsert_EmptySet_MarksEveryExistingRowRemoved()
|
||||
{
|
||||
await _storage.UpsertDeployedSecretsAsync([Art("A", 1), Art("B", 1)], "d1");
|
||||
|
||||
var (applied, removed) = await _storage.UpsertDeployedSecretsAsync([], "d2");
|
||||
|
||||
Assert.Empty(applied);
|
||||
Assert.Equal(["A", "B"], removed.OrderBy(n => n, StringComparer.Ordinal));
|
||||
|
||||
var rows = await _storage.GetDeployedSecretsAsync();
|
||||
Assert.All(rows, r => Assert.True(r.IsRemoved));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task GetDeployedSecretsAsync_RoundTripsEveryFieldOfTheSecretArtifact()
|
||||
{
|
||||
var art = Art("Full", 7);
|
||||
await _storage.UpsertDeployedSecretsAsync([art], "dep-full");
|
||||
|
||||
var row = Assert.Single(await _storage.GetDeployedSecretsAsync());
|
||||
|
||||
Assert.Equal("Full", row.Name);
|
||||
Assert.Equal("dep-full", row.DeploymentId);
|
||||
Assert.False(row.IsRemoved);
|
||||
Assert.Equal(art.Name, row.Secret.Name);
|
||||
Assert.Equal(art.ContentType, row.Secret.ContentType);
|
||||
Assert.Equal(art.Ciphertext, row.Secret.Ciphertext);
|
||||
Assert.Equal(art.Nonce, row.Secret.Nonce);
|
||||
Assert.Equal(art.Tag, row.Secret.Tag);
|
||||
Assert.Equal(art.WrappedDek, row.Secret.WrappedDek);
|
||||
Assert.Equal(art.WrapNonce, row.Secret.WrapNonce);
|
||||
Assert.Equal(art.WrapTag, row.Secret.WrapTag);
|
||||
Assert.Equal(art.KekId, row.Secret.KekId);
|
||||
Assert.Equal(art.Revision, row.Secret.Revision);
|
||||
Assert.Equal(art.CreatedUtc, row.Secret.CreatedUtc);
|
||||
Assert.Equal(art.UpdatedUtc, row.Secret.UpdatedUtc);
|
||||
Assert.Equal(art.Description, row.Secret.Description);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task GetDeployedSecretsAsync_IncludesRemovedRows()
|
||||
{
|
||||
await _storage.UpsertDeployedSecretsAsync([Art("A", 1)], "d1");
|
||||
await _storage.UpsertDeployedSecretsAsync([], "d2"); // A removed
|
||||
|
||||
var rows = await _storage.GetDeployedSecretsAsync();
|
||||
|
||||
var a = Assert.Single(rows);
|
||||
Assert.Equal("A", a.Name);
|
||||
Assert.True(a.IsRemoved);
|
||||
}
|
||||
|
||||
// ── security: row_json holds only base64 ciphertext, never plaintext ──
|
||||
|
||||
[Fact]
|
||||
public async Task RowJson_DeserializesToSecretArtifact_WithNoExtraProperties()
|
||||
{
|
||||
// Guards against ever widening SecretArtifact (or row_json) with a plaintext
|
||||
// field: this walks the RAW stored JSON, not the deserialized object, so a
|
||||
// stray extra property would be caught here even though JsonSerializer.Deserialize
|
||||
// silently ignores unknown members by default.
|
||||
var art = Art("Sec1", 3);
|
||||
await _storage.UpsertDeployedSecretsAsync([art], "dep-sec");
|
||||
|
||||
var rowJson = await ReadRawRowJsonAsync("Sec1");
|
||||
|
||||
// SecretArtifact is a record, but records only get member-wise Equals for reference
|
||||
// types via THEIR OWN Equals — and Array does not override Equals, so byte[] members
|
||||
// compare by reference. Assert.Equal(art, roundTripped) would therefore fail even on
|
||||
// a byte-for-byte-identical round trip; compare field-by-field instead.
|
||||
var roundTripped = JsonSerializer.Deserialize<SecretArtifact>(rowJson);
|
||||
Assert.NotNull(roundTripped);
|
||||
Assert.Equal(art.Name, roundTripped!.Name);
|
||||
Assert.Equal(art.ContentType, roundTripped.ContentType);
|
||||
Assert.Equal(art.Ciphertext, roundTripped.Ciphertext);
|
||||
Assert.Equal(art.Nonce, roundTripped.Nonce);
|
||||
Assert.Equal(art.Tag, roundTripped.Tag);
|
||||
Assert.Equal(art.WrappedDek, roundTripped.WrappedDek);
|
||||
Assert.Equal(art.WrapNonce, roundTripped.WrapNonce);
|
||||
Assert.Equal(art.WrapTag, roundTripped.WrapTag);
|
||||
Assert.Equal(art.KekId, roundTripped.KekId);
|
||||
Assert.Equal(art.Revision, roundTripped.Revision);
|
||||
Assert.Equal(art.CreatedUtc, roundTripped.CreatedUtc);
|
||||
Assert.Equal(art.UpdatedUtc, roundTripped.UpdatedUtc);
|
||||
Assert.Equal(art.Description, roundTripped.Description);
|
||||
|
||||
using var document = JsonDocument.Parse(rowJson);
|
||||
var propertyNames = document.RootElement.EnumerateObject()
|
||||
.Select(p => p.Name)
|
||||
.ToHashSet(StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
var expected = new HashSet<string>(StringComparer.OrdinalIgnoreCase)
|
||||
{
|
||||
nameof(SecretArtifact.Name), nameof(SecretArtifact.ContentType),
|
||||
nameof(SecretArtifact.Ciphertext), nameof(SecretArtifact.Nonce),
|
||||
nameof(SecretArtifact.Tag), nameof(SecretArtifact.WrappedDek),
|
||||
nameof(SecretArtifact.WrapNonce), nameof(SecretArtifact.WrapTag),
|
||||
nameof(SecretArtifact.KekId), nameof(SecretArtifact.Revision),
|
||||
nameof(SecretArtifact.CreatedUtc), nameof(SecretArtifact.UpdatedUtc),
|
||||
nameof(SecretArtifact.Description),
|
||||
};
|
||||
|
||||
Assert.Equal(expected, propertyNames);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task RowJson_EncodesEveryByteArrayField_AsBase64Text()
|
||||
{
|
||||
// byte[] serializes as base64 via System.Text.Json by default. Asserted at the raw
|
||||
// JSON level (JsonValueKind.String, not an array of numbers) so a future custom
|
||||
// converter that changed the wire shape would be caught here.
|
||||
var art = Art("Sec2", 1);
|
||||
await _storage.UpsertDeployedSecretsAsync([art], "dep-sec2");
|
||||
|
||||
var rowJson = await ReadRawRowJsonAsync("Sec2");
|
||||
using var document = JsonDocument.Parse(rowJson);
|
||||
var root = document.RootElement;
|
||||
|
||||
foreach (var byteArrayField in new[]
|
||||
{
|
||||
nameof(SecretArtifact.Ciphertext), nameof(SecretArtifact.Nonce),
|
||||
nameof(SecretArtifact.Tag), nameof(SecretArtifact.WrappedDek),
|
||||
nameof(SecretArtifact.WrapNonce), nameof(SecretArtifact.WrapTag),
|
||||
})
|
||||
{
|
||||
var property = root.GetProperty(byteArrayField);
|
||||
Assert.Equal(JsonValueKind.String, property.ValueKind);
|
||||
|
||||
// Base64.TryFromBase64String throws on malformed input — proves it really is
|
||||
// base64 text, not just any string.
|
||||
Assert.True(Convert.TryFromBase64String(property.GetString()!, new byte[64], out _));
|
||||
}
|
||||
}
|
||||
|
||||
private async Task<string> ReadRawRowJsonAsync(string name)
|
||||
{
|
||||
await using var connection = _storage.CreateConnection();
|
||||
await using var command = connection.CreateCommand();
|
||||
command.CommandText = "SELECT row_json FROM deployed_secrets WHERE name = @name";
|
||||
command.Parameters.AddWithValue("@name", name);
|
||||
return (string)(await command.ExecuteScalarAsync())!;
|
||||
}
|
||||
|
||||
// ── mirrored_secrets: node-local ledger ──
|
||||
|
||||
[Fact]
|
||||
public async Task MirroredSecrets_SetThenGet_RoundTrips()
|
||||
{
|
||||
await _storage.SetMirroredSecretAsync("Sec1", 3);
|
||||
|
||||
var rows = await _storage.GetMirroredSecretsAsync();
|
||||
|
||||
var row = Assert.Single(rows);
|
||||
Assert.Equal("Sec1", row.Name);
|
||||
Assert.Equal(3, row.Revision);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task MirroredSecrets_Set_IsUpsert()
|
||||
{
|
||||
await _storage.SetMirroredSecretAsync("Sec1", 1);
|
||||
await _storage.SetMirroredSecretAsync("Sec1", 2);
|
||||
|
||||
var rows = await _storage.GetMirroredSecretsAsync();
|
||||
|
||||
var row = Assert.Single(rows);
|
||||
Assert.Equal(2, row.Revision);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task MirroredSecrets_Remove_DeletesTheLedgerRow()
|
||||
{
|
||||
await _storage.SetMirroredSecretAsync("Sec1", 1);
|
||||
|
||||
await _storage.RemoveMirroredSecretAsync("Sec1");
|
||||
|
||||
Assert.Empty(await _storage.GetMirroredSecretsAsync());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task MirroredSecrets_Remove_OfAnUnknownName_IsANoOp()
|
||||
{
|
||||
await _storage.RemoveMirroredSecretAsync("NeverMirrored");
|
||||
|
||||
Assert.Empty(await _storage.GetMirroredSecretsAsync());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task MirroredSecrets_IsIndependentOf_DeployedSecrets()
|
||||
{
|
||||
// mirroring is a separate, node-local action from the site-wide deployed_secrets
|
||||
// reconcile — removing a deployed secret must not touch the mirrored ledger, and
|
||||
// vice versa. Mirroring/un-mirroring in response to a removal is the caller's job.
|
||||
await _storage.UpsertDeployedSecretsAsync([Art("A", 1)], "d1");
|
||||
await _storage.SetMirroredSecretAsync("A", 1);
|
||||
|
||||
await _storage.UpsertDeployedSecretsAsync([], "d2"); // A removed from deployed_secrets
|
||||
|
||||
var deployed = Assert.Single(await _storage.GetDeployedSecretsAsync());
|
||||
Assert.True(deployed.IsRemoved);
|
||||
|
||||
var mirrored = Assert.Single(await _storage.GetMirroredSecretsAsync());
|
||||
Assert.Equal("A", mirrored.Name);
|
||||
}
|
||||
|
||||
private static SecretArtifact Art(string name, long revision) => new(
|
||||
Name: name,
|
||||
ContentType: "application/octet-stream",
|
||||
Ciphertext: [1, 2, 3, 4, 5],
|
||||
Nonce: [10, 11, 12],
|
||||
Tag: [20, 21, 22, 23],
|
||||
WrappedDek: [30, 31, 32, 33, 34, 35],
|
||||
WrapNonce: [40, 41, 42],
|
||||
WrapTag: [50, 51, 52, 53],
|
||||
KekId: "kek-1",
|
||||
Revision: revision,
|
||||
CreatedUtc: new DateTimeOffset(2026, 9, 28, 0, 0, 0, TimeSpan.Zero),
|
||||
UpdatedUtc: new DateTimeOffset(2026, 9, 28, 1, 0, 0, TimeSpan.Zero),
|
||||
Description: "test secret");
|
||||
}
|
||||
@@ -23,13 +23,14 @@ public class SiteStorageSchemaTests
|
||||
SiteStorageSchema.Apply(connection);
|
||||
SiteStorageSchema.Apply(connection); // second run must not throw
|
||||
|
||||
// Every table is named explicitly on purpose — a loop asserting "9 tables"
|
||||
// Every table is named explicitly on purpose — a loop asserting "11 tables"
|
||||
// would still pass if one were renamed.
|
||||
foreach (var table in new[]
|
||||
{
|
||||
"deployed_configurations", "static_attribute_overrides", "shared_scripts",
|
||||
"external_systems", "database_connections", "notification_lists",
|
||||
"data_connection_definitions", "smtp_configurations", "native_alarm_state",
|
||||
"deployed_secrets", "mirrored_secrets",
|
||||
})
|
||||
{
|
||||
Assert.True(TableExists(connection, table), $"missing table: {table}");
|
||||
|
||||
+159
@@ -0,0 +1,159 @@
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.Audit;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Secrets;
|
||||
using ZB.MOM.WW.ScadaBridge.TestSupport;
|
||||
using ZB.MOM.WW.Secrets;
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
using ZB.MOM.WW.Secrets.Crypto;
|
||||
using ZB.MOM.WW.Secrets.Sqlite;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Secrets;
|
||||
|
||||
/// <summary>
|
||||
/// Task 9 fix round 2 (review <c>task-9-rereview-1.md</c> recommendation #2): runs
|
||||
/// <see cref="DeployedSecretsMirror"/> against the REAL <c>ZB.MOM.WW.Secrets</c> stack —
|
||||
/// <see cref="SqliteSecretStore"/> + <see cref="AesGcmEnvelopeCipher"/> +
|
||||
/// <see cref="DefaultSecretResolver"/> over a real temp-file SQLite database — instead of the
|
||||
/// in-memory fakes <see cref="DeployedSecretsMirrorTests"/> uses.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <see cref="DeployedSecretsMirrorTests"/>'s fakes never actually decrypt anything, so the
|
||||
/// class remarks' claim that a tombstone-recovery re-stamp (bumping <c>UpdatedUtc</c>/
|
||||
/// <c>Revision</c> without touching ciphertext) cannot break decryption was previously proven
|
||||
/// only by reading <c>AesGcmEnvelopeCipher</c>'s source (its AAD binds only
|
||||
/// <see cref="SecretName.Value"/> and <see cref="StoredSecret.KekId"/>, never
|
||||
/// <c>Revision</c>/<c>UpdatedUtc</c>). This test pins that property against the real cipher, so
|
||||
/// a future library version that starts binding either field into the AEAD associated data
|
||||
/// would fail here instead of only being caught by re-reading the source.
|
||||
/// </remarks>
|
||||
public sealed class DeployedSecretsMirrorRealStackTests : IAsyncLifetime, IDisposable
|
||||
{
|
||||
private readonly string _secretsDbPath =
|
||||
Path.Combine(Path.GetTempPath(), $"zb-secrets-real-{Guid.NewGuid():N}.db");
|
||||
private readonly TestLocalDb _localDb = TestLocalDb.CreateTemp("deployed-secrets-mirror-real");
|
||||
|
||||
private SqliteSecretStore _store = null!;
|
||||
private AesGcmEnvelopeCipher _cipher = null!;
|
||||
private DefaultSecretResolver _resolver = null!;
|
||||
private SiteStorageService _storage = null!;
|
||||
|
||||
public async Task InitializeAsync()
|
||||
{
|
||||
var connectionFactory = new SecretsSqliteConnectionFactory(_secretsDbPath);
|
||||
await new SqliteSecretsStoreMigrator(connectionFactory).MigrateAsync(CancellationToken.None);
|
||||
_store = new SqliteSecretStore(connectionFactory);
|
||||
|
||||
var masterKey = new FakeMasterKeyProvider("kek-central");
|
||||
_cipher = new AesGcmEnvelopeCipher(masterKey);
|
||||
// DefaultSecretResolver implements BOTH ISecretResolver and ISecretCacheInvalidator (the
|
||||
// real production shape — see SecretsServiceCollectionExtensions, which registers the
|
||||
// same singleton instance behind both interfaces), so this test also exercises real
|
||||
// cache invalidation, not a fake one.
|
||||
_resolver = new DefaultSecretResolver(_store, _cipher, NoOpAuditWriter.Instance, TimeSpan.FromSeconds(30));
|
||||
|
||||
_storage = new SiteStorageService(_localDb.Db, NullLogger<SiteStorageService>.Instance);
|
||||
await _storage.InitializeAsync();
|
||||
}
|
||||
|
||||
public Task DisposeAsync() => Task.CompletedTask;
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
var path = _localDb.Path;
|
||||
_localDb.Dispose();
|
||||
TestLocalDb.DeleteFiles(path);
|
||||
foreach (var suffix in new[] { "", "-wal", "-shm" })
|
||||
{
|
||||
try { File.Delete(_secretsDbPath + suffix); } catch { /* best effort */ }
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task RealStack_EncryptApplyDeleteReAdd_TheRecoveredRowResolvesToTheOriginalPlaintext()
|
||||
{
|
||||
const string plaintext = "s3cr3t-value";
|
||||
|
||||
// "Central" encrypts the secret under the fleet's shared KEK (the same KekId this
|
||||
// node's own master key provider reports, mirroring a real shared-KEK fleet) and ships
|
||||
// it as a SecretArtifact — the exact wire shape DeployedSecretsCollector produces.
|
||||
var sealedSecret = _cipher.Encrypt(new SecretName("a"), plaintext, SecretContentType.Text);
|
||||
// Deliberately far in the past so DeleteAsync's real-wall-clock tombstone stamp is
|
||||
// guaranteed strictly newer, regardless of when this test runs — the point being
|
||||
// tested is the RECOVERY, not a race against the tombstone.
|
||||
var shippedUtc = new DateTimeOffset(2000, 1, 1, 0, 0, 0, TimeSpan.Zero);
|
||||
var artifact = new SecretArtifact(
|
||||
Name: "a",
|
||||
ContentType: SecretContentType.Text.ToString(),
|
||||
Ciphertext: sealedSecret.Ciphertext,
|
||||
Nonce: sealedSecret.Nonce,
|
||||
Tag: sealedSecret.Tag,
|
||||
WrappedDek: sealedSecret.WrappedDek,
|
||||
WrapNonce: sealedSecret.WrapNonce,
|
||||
WrapTag: sealedSecret.WrapTag,
|
||||
KekId: sealedSecret.KekId,
|
||||
Revision: 1,
|
||||
CreatedUtc: shippedUtc,
|
||||
UpdatedUtc: shippedUtc);
|
||||
|
||||
var mirror = new DeployedSecretsMirror(
|
||||
_storage,
|
||||
_store,
|
||||
_resolver,
|
||||
_resolver, // the same singleton also serves ISecretCacheInvalidator
|
||||
Options.Create(new SiteRuntimeOptions()),
|
||||
NullLogger<DeployedSecretsMirror>.Instance,
|
||||
TimeProvider.System);
|
||||
|
||||
// Apply — a fresh name, so this is a plain (non-recovery) apply.
|
||||
await _storage.UpsertDeployedSecretsAsync([artifact], "d1");
|
||||
var applyPass = await mirror.ReconcileAsync(CancellationToken.None);
|
||||
Assert.Equal(["a"], applyPass.Applied);
|
||||
Assert.Empty(applyPass.Failed);
|
||||
Assert.Equal(plaintext, await _resolver.GetAsync(new SecretName("a"), CancellationToken.None));
|
||||
|
||||
// Delete — the site stops referencing it; the REAL store tombstones (does not
|
||||
// hard-delete) at DateTimeOffset.UtcNow, which is far newer than the 2000-01-01 shipped
|
||||
// stamp above.
|
||||
await _storage.UpsertDeployedSecretsAsync([], "d2");
|
||||
var removePass = await mirror.ReconcileAsync(CancellationToken.None);
|
||||
Assert.Equal(["a"], removePass.Removed);
|
||||
Assert.Null(await _resolver.GetAsync(new SecretName("a"), CancellationToken.None));
|
||||
|
||||
// Re-add the SAME (unchanged, still 2000-01-01-stamped) artifact. A naive
|
||||
// ApplyReplicatedAsync call is a no-op against the newer tombstone — this forces the
|
||||
// mirror's tombstone-recovery re-stamp path against the REAL store and REAL cipher.
|
||||
await _storage.UpsertDeployedSecretsAsync([artifact], "d3");
|
||||
var reAddPass = await mirror.ReconcileAsync(CancellationToken.None);
|
||||
Assert.Equal(["a"], reAddPass.Applied);
|
||||
Assert.Empty(reAddPass.Failed);
|
||||
|
||||
// The recovered row must decrypt to the ORIGINAL plaintext through the real
|
||||
// AesGcmEnvelopeCipher, proving the bumped UpdatedUtc/Revision are not — and cannot
|
||||
// silently become — bound into the AEAD associated data.
|
||||
var resolved = await _resolver.GetAsync(new SecretName("a"), CancellationToken.None);
|
||||
Assert.Equal(plaintext, resolved);
|
||||
|
||||
var recovered = await _store.GetAsync(new SecretName("a"), CancellationToken.None);
|
||||
Assert.NotNull(recovered);
|
||||
Assert.False(recovered!.IsDeleted);
|
||||
Assert.True(recovered.UpdatedUtc > shippedUtc); // genuinely re-stamped, not a no-op
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A hand-written <see cref="IMasterKeyProvider"/> test double, mirroring the library's own
|
||||
/// (test-internal, not shipped in the package) <c>FakeMasterKeyProvider</c>.
|
||||
/// </summary>
|
||||
private sealed class FakeMasterKeyProvider : IMasterKeyProvider
|
||||
{
|
||||
private readonly byte[] _key = System.Security.Cryptography.RandomNumberGenerator.GetBytes(32);
|
||||
|
||||
public FakeMasterKeyProvider(string kekId) => KekId = kekId;
|
||||
|
||||
public string KekId { get; }
|
||||
|
||||
public ReadOnlyMemory<byte> GetMasterKey() => _key;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,894 @@
|
||||
using System.Collections.Concurrent;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Secrets;
|
||||
using ZB.MOM.WW.ScadaBridge.TestSupport;
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Secrets;
|
||||
|
||||
/// <summary>
|
||||
/// Task 9, site-secrets artifact deploy: <see cref="DeployedSecretsMirror"/> — per-node apply
|
||||
/// + decrypt-verify of <c>deployed_secrets</c> into this node's own secret store.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Fix round 1 (review <c>task-9-review.md</c>): <see cref="FakeSecretStore"/> now honours the
|
||||
/// real <c>SqliteSecretStore</c> semantics — <c>ApplyReplicatedAsync</c> is strict
|
||||
/// last-writer-wins via <see cref="SecretLastWriterWins.IsNewer"/>, and <c>DeleteAsync</c>
|
||||
/// tombstones rather than hard-deleting — so tests exercise the same conflict shapes the real
|
||||
/// store can produce, not an idealized "always overwrite" double.
|
||||
/// </remarks>
|
||||
public class DeployedSecretsMirrorTests : IDisposable
|
||||
{
|
||||
private readonly TestLocalDb _localDb;
|
||||
|
||||
public DeployedSecretsMirrorTests()
|
||||
{
|
||||
_localDb = TestLocalDb.CreateTemp("deployed-secrets-mirror");
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
var path = _localDb.Path;
|
||||
_localDb.Dispose();
|
||||
TestLocalDb.DeleteFiles(path);
|
||||
}
|
||||
|
||||
// ── Apply / remove / out-of-band ──
|
||||
|
||||
[Fact]
|
||||
public async Task Reconcile_Applies_Verifies_AndOnlyDeletesWhatItMirrored()
|
||||
{
|
||||
var (mirror, storage, store, _, _) = await CreateAsync();
|
||||
// Seeded OLDER than what will be shipped, so the shipped row legitimately wins LWW
|
||||
// and is applied — the strictly-NEWER out-of-band case is covered separately by
|
||||
// Reconcile_OutOfBandNewerSecret_NotOverwritten_NotLedgered_AndSurvivesLaterRemoval.
|
||||
store.Seed("out-of-band", updatedUtc: new DateTimeOffset(2026, 9, 27, 0, 0, 0, TimeSpan.Zero));
|
||||
|
||||
await storage.UpsertDeployedSecretsAsync([Art("a", 1), Art("out-of-band", 1)], "d1");
|
||||
var r1 = await mirror.ReconcileAsync(CancellationToken.None);
|
||||
|
||||
Assert.Equal(new[] { "a", "out-of-band" }, r1.Applied.OrderBy(x => x, StringComparer.Ordinal));
|
||||
Assert.Empty(r1.Failed);
|
||||
|
||||
await storage.UpsertDeployedSecretsAsync([], "d2");
|
||||
// Simulate: "out-of-band" was never actually mirrored by this node (an operator secret
|
||||
// that happens to share the name) by clearing its ledger row directly — a defensive
|
||||
// check independent of the natural not-ledgered path exercised below.
|
||||
await storage.RemoveMirroredSecretAsync("out-of-band");
|
||||
|
||||
var r2 = await mirror.ReconcileAsync(CancellationToken.None);
|
||||
|
||||
Assert.Equal(["a"], r2.Removed);
|
||||
Assert.True(store.Contains("out-of-band"));
|
||||
Assert.False(store.Contains("a"));
|
||||
Assert.True(store.IsTombstoned("a"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Reconcile_OutOfBandNewerSecret_NotOverwritten_NotLedgered_AndSurvivesLaterRemoval()
|
||||
{
|
||||
var (mirror, storage, store, _, invalidator) = await CreateAsync();
|
||||
var operatorCiphertext = new byte[] { 200, 201, 202 };
|
||||
// Strictly newer than Art()'s fixed UpdatedUtc (2026-09-28T01:00:00Z) — legitimately
|
||||
// wins LWW, exactly as the real store would order it.
|
||||
store.Seed("a",
|
||||
updatedUtc: new DateTimeOffset(2026, 9, 28, 2, 0, 0, TimeSpan.Zero),
|
||||
revision: 5,
|
||||
ciphertext: operatorCiphertext);
|
||||
|
||||
await storage.UpsertDeployedSecretsAsync([Art("a", 1)], "d1");
|
||||
var r1 = await mirror.ReconcileAsync(CancellationToken.None);
|
||||
|
||||
Assert.Empty(r1.Applied);
|
||||
var message = Assert.Single(r1.Failed);
|
||||
Assert.Contains("'a'", message);
|
||||
Assert.Contains("newer local value", message);
|
||||
Assert.DoesNotContain("did not decrypt", message); // distinct from the master-key message
|
||||
Assert.Equal(operatorCiphertext, store.Rows["a"].Ciphertext); // untouched
|
||||
Assert.Empty(invalidator.Invalidated); // never applied -> never invalidated
|
||||
|
||||
// Never ledgered, so a later central tombstone must NOT delete the operator's secret.
|
||||
await storage.UpsertDeployedSecretsAsync([], "d2");
|
||||
var r2 = await mirror.ReconcileAsync(CancellationToken.None);
|
||||
|
||||
Assert.Empty(r2.Removed);
|
||||
Assert.True(store.Contains("a"));
|
||||
Assert.Equal(operatorCiphertext, store.Rows["a"].Ciphertext);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Reconcile_RemoveThenReAdd_TombstoneIsRecovered_ReappliesAndVerifies()
|
||||
{
|
||||
var time = new FakeTimeProvider(new DateTimeOffset(2026, 9, 28, 6, 0, 0, TimeSpan.Zero));
|
||||
var (mirror, storage, store, _, invalidator) = await CreateAsync(time: time);
|
||||
await storage.UpsertDeployedSecretsAsync([Art("a", 1)], "d1");
|
||||
var firstApply = await mirror.ReconcileAsync(CancellationToken.None);
|
||||
Assert.Equal(["a"], firstApply.Applied);
|
||||
|
||||
// Site stops referencing it: central tombstones the row, this node deletes it. The
|
||||
// tombstone is stamped LATER than the originally-shipped UpdatedUtc (01:00).
|
||||
store.Now = new DateTimeOffset(2026, 9, 28, 12, 0, 0, TimeSpan.Zero);
|
||||
await storage.UpsertDeployedSecretsAsync([], "d2");
|
||||
var removePass = await mirror.ReconcileAsync(CancellationToken.None);
|
||||
Assert.Equal(["a"], removePass.Removed);
|
||||
Assert.True(store.IsTombstoned("a"));
|
||||
invalidator.Invalidated.Clear();
|
||||
|
||||
// Site references it again — central ships the SAME (unchanged, older-stamped)
|
||||
// secret. A naive ApplyReplicatedAsync call is a no-op against the newer tombstone;
|
||||
// the mirror must detect and recover this.
|
||||
time.Now = new DateTimeOffset(2026, 9, 28, 18, 0, 0, TimeSpan.Zero); // later than the tombstone
|
||||
await storage.UpsertDeployedSecretsAsync([Art("a", 1)], "d3");
|
||||
var reAddPass = await mirror.ReconcileAsync(CancellationToken.None);
|
||||
|
||||
Assert.Equal(["a"], reAddPass.Applied);
|
||||
Assert.Empty(reAddPass.Failed);
|
||||
Assert.True(store.Contains("a"));
|
||||
Assert.Equal(new byte[] { 1, 2, 3, 4, 5 }, store.Rows["a"].Ciphertext); // Art()'s ciphertext, re-materialized
|
||||
Assert.Contains("a", invalidator.Invalidated);
|
||||
|
||||
// Ledgering used the SHIPPED artifact's own revision, not the bumped local one, so a
|
||||
// later pass with the same shipped row is a true no-op.
|
||||
var thirdPass = await mirror.ReconcileAsync(CancellationToken.None);
|
||||
Assert.Empty(thirdPass.Applied);
|
||||
Assert.False(thirdPass.Changed);
|
||||
|
||||
var mirrored = await storage.GetMirroredSecretsAsync();
|
||||
Assert.Equal(1, mirrored.Single(m => m.Name == "a").Revision);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Reconcile_ClockSkewedRecoveredRow_StillConvergesOnNextCentralRevision()
|
||||
{
|
||||
// Fix round 2 (review "clock skew" finding): a tombstone-recovery bump can leave a local
|
||||
// row stamped AHEAD of a central UpdatedUtc that arrives afterward (site clock skew, or
|
||||
// simply a central edit made before the recovery but shipped after it). The row is still
|
||||
// OWNED by this mirror's ledger, so a later genuinely-new central revision must not be
|
||||
// misreported as an out-of-band "newer local value" — it must converge.
|
||||
var time = new FakeTimeProvider(new DateTimeOffset(2026, 9, 28, 6, 0, 0, TimeSpan.Zero));
|
||||
var (mirror, storage, store, _, invalidator) = await CreateAsync(time: time);
|
||||
|
||||
await storage.UpsertDeployedSecretsAsync([Art("a", 1)], "d1");
|
||||
await mirror.ReconcileAsync(CancellationToken.None);
|
||||
|
||||
// Remove then re-add — but the SITE CLOCK IS SKEWED FAR INTO THE FUTURE while the
|
||||
// recovery bump happens (a bad RTC/NTP fault), so the recovered row's UpdatedUtc lands
|
||||
// far ahead of anything central will ship for a long while.
|
||||
store.Now = new DateTimeOffset(2026, 9, 28, 12, 0, 0, TimeSpan.Zero);
|
||||
await storage.UpsertDeployedSecretsAsync([], "d2");
|
||||
await mirror.ReconcileAsync(CancellationToken.None); // tombstones "a"
|
||||
|
||||
time.Now = new DateTimeOffset(2099, 1, 1, 0, 0, 0, TimeSpan.Zero); // skewed site clock
|
||||
await storage.UpsertDeployedSecretsAsync([Art("a", 1)], "d3");
|
||||
var recoveredPass = await mirror.ReconcileAsync(CancellationToken.None);
|
||||
Assert.Equal(["a"], recoveredPass.Applied);
|
||||
Assert.True(store.Rows["a"].UpdatedUtc.Year >= 2099); // confirms the skewed stamp landed
|
||||
|
||||
// Site clock recovers to normal. Central ships a genuinely NEW revision (different
|
||||
// ciphertext, i.e. an actual rotation) with an ordinary timestamp — OLDER than the
|
||||
// skewed local stamp, so a naive ApplyReplicatedAsync would no-op against it.
|
||||
time.Now = new DateTimeOffset(2026, 9, 29, 0, 0, 0, TimeSpan.Zero);
|
||||
var rotatedCiphertext = new byte[] { 7, 7, 7 };
|
||||
await storage.UpsertDeployedSecretsAsync([Art("a", 2, ciphertext: rotatedCiphertext)], "d4");
|
||||
|
||||
var convergePass = await mirror.ReconcileAsync(CancellationToken.None);
|
||||
|
||||
Assert.Equal(["a"], convergePass.Applied);
|
||||
Assert.Empty(convergePass.Failed); // NOT "newer local value" — this mirror owns the name
|
||||
Assert.Equal(rotatedCiphertext, store.Rows["a"].Ciphertext);
|
||||
Assert.Contains("a", invalidator.Invalidated);
|
||||
|
||||
var mirrored = await storage.GetMirroredSecretsAsync();
|
||||
Assert.Equal(2, mirrored.Single(m => m.Name == "a").Revision);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Reconcile_ReAppliesOnlyWhenRevisionChanges()
|
||||
{
|
||||
var (mirror, storage, store, _, _) = await CreateAsync();
|
||||
|
||||
await storage.UpsertDeployedSecretsAsync([Art("a", 1)], "d1");
|
||||
var r1 = await mirror.ReconcileAsync(CancellationToken.None);
|
||||
Assert.Equal(["a"], r1.Applied);
|
||||
|
||||
var r2 = await mirror.ReconcileAsync(CancellationToken.None);
|
||||
Assert.Empty(r2.Applied);
|
||||
Assert.False(r2.Changed);
|
||||
|
||||
await storage.UpsertDeployedSecretsAsync([Art("a", 2)], "d2");
|
||||
var r3 = await mirror.ReconcileAsync(CancellationToken.None);
|
||||
Assert.Equal(["a"], r3.Applied);
|
||||
Assert.Equal(2, store.Rows["a"].Revision);
|
||||
}
|
||||
|
||||
// ── Same-revision drift (final review I1) ──
|
||||
|
||||
[Fact]
|
||||
public async Task Reconcile_OwnedNameDeletedLocally_IsHealedAndVerified_NotMisreportedAsKekMismatch()
|
||||
{
|
||||
var time = new FakeTimeProvider(new DateTimeOffset(2026, 9, 28, 18, 0, 0, TimeSpan.Zero));
|
||||
var (mirror, storage, store, resolver, invalidator) = await CreateAsync(time: time);
|
||||
resolver.BackingStore = store; // resolve returns null once the local row is gone
|
||||
await storage.UpsertDeployedSecretsAsync([Art("a", 1)], "d1");
|
||||
Assert.Equal(["a"], (await mirror.ReconcileAsync(CancellationToken.None)).Applied);
|
||||
|
||||
// An operator deletes the mirror-owned name on this node (library CLI/UI) — the
|
||||
// ledger still says r1 is mirrored, and central still ships r1.
|
||||
await store.DeleteAsync(new SecretName("a"), "operator", CancellationToken.None);
|
||||
Assert.True(store.IsTombstoned("a"));
|
||||
invalidator.Invalidated.Clear();
|
||||
|
||||
var r = await mirror.ReconcileAsync(CancellationToken.None);
|
||||
|
||||
Assert.Empty(r.Failed);
|
||||
Assert.Equal(["a"], r.Applied);
|
||||
Assert.True(store.Contains("a"));
|
||||
Assert.Equal(new byte[] { 1, 2, 3, 4, 5 }, store.Rows["a"].Ciphertext);
|
||||
Assert.Contains("a", invalidator.Invalidated);
|
||||
|
||||
// Healed state is stable: the next pass is a no-op that still verifies clean.
|
||||
var next = await mirror.ReconcileAsync(CancellationToken.None);
|
||||
Assert.False(next.Changed);
|
||||
Assert.Empty(next.Failed);
|
||||
Assert.Equal(1, (await storage.GetMirroredSecretsAsync()).Single(m => m.Name == "a").Revision);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Reconcile_OwnedNameEditedLocally_IsRevertedToTheDeployedCiphertext()
|
||||
{
|
||||
var time = new FakeTimeProvider(new DateTimeOffset(2026, 9, 28, 18, 0, 0, TimeSpan.Zero));
|
||||
var (mirror, storage, store, _, invalidator) = await CreateAsync(time: time);
|
||||
await storage.UpsertDeployedSecretsAsync([Art("a", 1)], "d1");
|
||||
Assert.Equal(["a"], (await mirror.ReconcileAsync(CancellationToken.None)).Applied);
|
||||
|
||||
// An operator overwrites the mirror-owned name locally with a NEWER stamp.
|
||||
store.Seed("a",
|
||||
updatedUtc: new DateTimeOffset(2026, 9, 28, 12, 0, 0, TimeSpan.Zero),
|
||||
revision: 7,
|
||||
ciphertext: [200, 201, 202]);
|
||||
invalidator.Invalidated.Clear();
|
||||
|
||||
var r = await mirror.ReconcileAsync(CancellationToken.None);
|
||||
|
||||
Assert.Empty(r.Failed);
|
||||
Assert.Equal(["a"], r.Applied);
|
||||
Assert.Equal(new byte[] { 1, 2, 3, 4, 5 }, store.Rows["a"].Ciphertext);
|
||||
Assert.Equal(new byte[] { 30, 31, 32, 33, 34, 35 }, store.Rows["a"].WrappedDek);
|
||||
Assert.True(store.Rows["a"].Revision > 7); // re-stamped past the operator's row
|
||||
Assert.Contains("a", invalidator.Invalidated);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Reconcile_OwnedNameDrift_LogsOncePerDriftNotEveryPass()
|
||||
{
|
||||
var logger = new CapturingLogger<DeployedSecretsMirror>();
|
||||
var time = new FakeTimeProvider(new DateTimeOffset(2026, 9, 28, 18, 0, 0, TimeSpan.Zero));
|
||||
var (mirror, storage, store, _, _) = await CreateAsync(logger: logger, time: time);
|
||||
await storage.UpsertDeployedSecretsAsync([Art("a", 1)], "d1");
|
||||
await mirror.ReconcileAsync(CancellationToken.None);
|
||||
await store.DeleteAsync(new SecretName("a"), "operator", CancellationToken.None);
|
||||
|
||||
await mirror.ReconcileAsync(CancellationToken.None);
|
||||
await mirror.ReconcileAsync(CancellationToken.None);
|
||||
await mirror.ReconcileAsync(CancellationToken.None);
|
||||
|
||||
Assert.Single(logger.Entries, e => e.Level == LogLevel.Warning && e.Message.Contains("re-applying"));
|
||||
Assert.DoesNotContain(logger.Entries, e => e.Level == LogLevel.Error);
|
||||
}
|
||||
|
||||
// ── Unreadable row_json (final review M4) ──
|
||||
|
||||
[Fact]
|
||||
public async Task Reconcile_UnparseableRowJson_IsAFailedEntryForThatName_OthersStillApply()
|
||||
{
|
||||
var (mirror, storage, store, _, _) = await CreateAsync();
|
||||
await storage.UpsertDeployedSecretsAsync([Art("bad", 1), Art("good", 1)], "d1");
|
||||
await CorruptRowJsonAsync(storage, "bad");
|
||||
|
||||
var r = await mirror.ReconcileAsync(CancellationToken.None);
|
||||
|
||||
Assert.Equal(["good"], r.Applied);
|
||||
Assert.Equal(["bad"], r.FailedNames);
|
||||
var message = Assert.Single(r.Failed);
|
||||
Assert.Contains("'bad'", message);
|
||||
Assert.Contains("could not be read", message);
|
||||
Assert.True(store.Contains("good"));
|
||||
Assert.False(store.Contains("bad"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Reconcile_UnparseableRemovedRow_StillRemovesAMirroredName()
|
||||
{
|
||||
var (mirror, storage, store, _, _) = await CreateAsync();
|
||||
await storage.UpsertDeployedSecretsAsync([Art("a", 1)], "d1");
|
||||
await mirror.ReconcileAsync(CancellationToken.None);
|
||||
await storage.UpsertDeployedSecretsAsync([], "d2"); // soft-removes "a"
|
||||
await CorruptRowJsonAsync(storage, "a");
|
||||
|
||||
var r = await mirror.ReconcileAsync(CancellationToken.None);
|
||||
|
||||
Assert.Equal(["a"], r.Removed);
|
||||
Assert.Empty(r.Failed);
|
||||
Assert.True(store.IsTombstoned("a"));
|
||||
}
|
||||
|
||||
private static async Task CorruptRowJsonAsync(SiteStorageService storage, string name)
|
||||
{
|
||||
await using var connection = storage.CreateConnection();
|
||||
await using var command = connection.CreateCommand();
|
||||
command.CommandText = "UPDATE deployed_secrets SET row_json = '{not json' WHERE name = @name";
|
||||
command.Parameters.AddWithValue("@name", name);
|
||||
Assert.Equal(1, await command.ExecuteNonQueryAsync());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Reconcile_AppliesThroughApplyReplicatedAsync_MappedFromSecretArtifact()
|
||||
{
|
||||
var (mirror, storage, store, _, _) = await CreateAsync();
|
||||
await storage.UpsertDeployedSecretsAsync([Art("a", 1, kek: "kek-central")], "d1");
|
||||
|
||||
await mirror.ReconcileAsync(CancellationToken.None);
|
||||
|
||||
var applied = store.Rows["a"];
|
||||
Assert.Equal("kek-central", applied.KekId);
|
||||
Assert.Equal(1, applied.Revision);
|
||||
Assert.Equal(SecretContentType.Text, applied.ContentType);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(SecretContentType.Text)]
|
||||
[InlineData(SecretContentType.ConnectionString)]
|
||||
[InlineData(SecretContentType.Json)]
|
||||
[InlineData(SecretContentType.BinaryBase64)]
|
||||
public async Task Reconcile_RoundTripsEveryContentType(SecretContentType contentType)
|
||||
{
|
||||
var (mirror, storage, store, _, _) = await CreateAsync();
|
||||
await storage.UpsertDeployedSecretsAsync(
|
||||
[Art("a", 1, contentType: contentType.ToString())], "d1");
|
||||
|
||||
await mirror.ReconcileAsync(CancellationToken.None);
|
||||
|
||||
Assert.Equal(contentType, store.Rows["a"].ContentType);
|
||||
}
|
||||
|
||||
// ── Per-row isolation (ruling 2) ──
|
||||
|
||||
[Fact]
|
||||
public async Task Reconcile_OneRowThrows_OthersStillApply()
|
||||
{
|
||||
var (mirror, storage, store, _, _) = await CreateAsync();
|
||||
// "bad" carries an unrecognised ContentType (as a newer central under version skew
|
||||
// might ship), so ToStored's Enum.Parse throws for that row only.
|
||||
await storage.UpsertDeployedSecretsAsync(
|
||||
[Art("bad", 1, contentType: "NotARealContentType"), Art("good", 1)], "d1");
|
||||
|
||||
var r = await mirror.ReconcileAsync(CancellationToken.None);
|
||||
|
||||
Assert.Equal(["good"], r.Applied);
|
||||
var message = Assert.Single(r.Failed);
|
||||
Assert.Contains("'bad'", message);
|
||||
Assert.True(store.Contains("good"));
|
||||
Assert.False(store.Contains("bad"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Reconcile_OneRowThrows_FailureIsDeSpammedLikeAnyOtherFailure()
|
||||
{
|
||||
var logger = new CapturingLogger<DeployedSecretsMirror>();
|
||||
var (mirror, storage, _, _, _) = await CreateAsync(logger);
|
||||
await storage.UpsertDeployedSecretsAsync([Art("bad", 1, contentType: "NotARealContentType")], "d1");
|
||||
|
||||
await mirror.ReconcileAsync(CancellationToken.None);
|
||||
await mirror.ReconcileAsync(CancellationToken.None);
|
||||
|
||||
var levels = logger.Entries
|
||||
.Where(e => e.Message.Contains("'bad'") && e.Message.Contains("failed to process"))
|
||||
.Select(e => e.Level)
|
||||
.ToList();
|
||||
Assert.Equal([LogLevel.Error, LogLevel.Debug], levels);
|
||||
}
|
||||
|
||||
// ── Cache invalidation (ruling 3) ──
|
||||
|
||||
[Fact]
|
||||
public async Task Reconcile_SuccessfulApply_InvalidatesCache_BeforeVerify()
|
||||
{
|
||||
var (mirror, storage, _, _, invalidator) = await CreateAsync();
|
||||
await storage.UpsertDeployedSecretsAsync([Art("a", 1)], "d1");
|
||||
|
||||
await mirror.ReconcileAsync(CancellationToken.None);
|
||||
|
||||
Assert.Contains("a", invalidator.Invalidated);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Reconcile_Removal_InvalidatesCache()
|
||||
{
|
||||
var (mirror, storage, _, _, invalidator) = await CreateAsync();
|
||||
await storage.UpsertDeployedSecretsAsync([Art("a", 1)], "d1");
|
||||
await mirror.ReconcileAsync(CancellationToken.None);
|
||||
invalidator.Invalidated.Clear();
|
||||
|
||||
await storage.UpsertDeployedSecretsAsync([], "d2");
|
||||
await mirror.ReconcileAsync(CancellationToken.None);
|
||||
|
||||
Assert.Contains("a", invalidator.Invalidated);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Reconcile_WithNoInvalidatorRegistered_StillApplies()
|
||||
{
|
||||
var storage = new SiteStorageService(_localDb.Db, NullLogger<SiteStorageService>.Instance);
|
||||
await storage.InitializeAsync();
|
||||
var mirror = new DeployedSecretsMirror(
|
||||
storage,
|
||||
new FakeSecretStore(),
|
||||
new FakeSecretResolver(),
|
||||
invalidator: null,
|
||||
Options.Create(new SiteRuntimeOptions { DeployedSecretsMirrorInterval = TimeSpan.FromSeconds(15) }),
|
||||
NullLogger<DeployedSecretsMirror>.Instance,
|
||||
TimeProvider.System);
|
||||
|
||||
await storage.UpsertDeployedSecretsAsync([Art("a", 1)], "d1");
|
||||
var r = await mirror.ReconcileAsync(CancellationToken.None);
|
||||
|
||||
Assert.Equal(["a"], r.Applied);
|
||||
}
|
||||
|
||||
// ── Decrypt verification ──
|
||||
|
||||
[Fact]
|
||||
public async Task Reconcile_DecryptFailure_ExceptionFromResolver_IsReportedWithKekId()
|
||||
{
|
||||
var (mirror, storage, _, resolver, _) = await CreateAsync();
|
||||
resolver.ThrowFor("a");
|
||||
|
||||
await storage.UpsertDeployedSecretsAsync([Art("a", 1, kek: "kek-central")], "d1");
|
||||
var r = await mirror.ReconcileAsync(CancellationToken.None);
|
||||
|
||||
var message = Assert.Single(r.Failed);
|
||||
Assert.Contains("kek-central", message);
|
||||
Assert.Contains("'a'", message);
|
||||
// FailedNames (Task 10): the name behind each Failed entry, in lockstep order, so a
|
||||
// caller (DeploymentManagerActor) can tell whether a failure belongs to names it shipped.
|
||||
Assert.Equal(["a"], r.FailedNames);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Reconcile_DecryptFailure_NullResolve_IsAlsoReported()
|
||||
{
|
||||
var (mirror, storage, _, resolver, _) = await CreateAsync();
|
||||
resolver.ReturnNullFor("a");
|
||||
|
||||
await storage.UpsertDeployedSecretsAsync([Art("a", 1)], "d1");
|
||||
var r = await mirror.ReconcileAsync(CancellationToken.None);
|
||||
|
||||
Assert.Single(r.Failed);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Reconcile_VerifiesEveryPass_EvenWhenAlreadyApplied()
|
||||
{
|
||||
var (mirror, storage, _, resolver, _) = await CreateAsync();
|
||||
await storage.UpsertDeployedSecretsAsync([Art("a", 1)], "d1");
|
||||
|
||||
var r1 = await mirror.ReconcileAsync(CancellationToken.None);
|
||||
Assert.Empty(r1.Failed);
|
||||
|
||||
// Master key "changes" (or a KEK rotation invalidates the local unwrap) between
|
||||
// passes — even though nothing about deployed_secrets changed, the next reconcile
|
||||
// must still catch the failure because it re-verifies unconditionally.
|
||||
resolver.ThrowFor("a");
|
||||
var r2 = await mirror.ReconcileAsync(CancellationToken.None);
|
||||
Assert.Single(r2.Failed);
|
||||
Assert.Empty(r2.Applied); // unchanged revision — not re-applied, only re-verified
|
||||
}
|
||||
|
||||
// ── Error-log de-spam (ruling 1 of the original task, unaffected by fix round 1) ──
|
||||
|
||||
[Fact]
|
||||
public async Task Reconcile_DecryptFailure_LogsErrorOnlyFirstTime_ThenDebug_ThenErrorAgainAfterASuccess()
|
||||
{
|
||||
var logger = new CapturingLogger<DeployedSecretsMirror>();
|
||||
var (mirror, storage, _, resolver, _) = await CreateAsync(logger);
|
||||
resolver.ThrowFor("a");
|
||||
await storage.UpsertDeployedSecretsAsync([Art("a", 1, kek: "kek-1")], "d1");
|
||||
|
||||
await mirror.ReconcileAsync(CancellationToken.None); // 1st failure: Error
|
||||
await mirror.ReconcileAsync(CancellationToken.None); // 2nd identical failure: Debug
|
||||
await mirror.ReconcileAsync(CancellationToken.None); // 3rd identical failure: Debug
|
||||
|
||||
var failureEntries = logger.Entries
|
||||
.Where(e => e.Message.Contains("did not decrypt"))
|
||||
.ToList();
|
||||
Assert.Equal(3, failureEntries.Count);
|
||||
Assert.Equal(LogLevel.Error, failureEntries[0].Level);
|
||||
Assert.Equal(LogLevel.Debug, failureEntries[1].Level);
|
||||
Assert.Equal(LogLevel.Debug, failureEntries[2].Level);
|
||||
|
||||
// Now the secret resolves (e.g. the key issue is fixed).
|
||||
resolver.ClearFailures();
|
||||
var recovered = await mirror.ReconcileAsync(CancellationToken.None);
|
||||
Assert.Empty(recovered.Failed);
|
||||
|
||||
// Fails again — must log at Error again, not Debug, since it succeeded in between.
|
||||
resolver.ThrowFor("a");
|
||||
await mirror.ReconcileAsync(CancellationToken.None);
|
||||
|
||||
var failureEntriesAfterRecovery = logger.Entries
|
||||
.Where(e => e.Message.Contains("did not decrypt"))
|
||||
.ToList();
|
||||
Assert.Equal(4, failureEntriesAfterRecovery.Count);
|
||||
Assert.Equal(LogLevel.Error, failureEntriesAfterRecovery[3].Level);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Reconcile_DecryptFailure_NewRevision_LogsErrorAgain_EvenWithoutASuccessInBetween()
|
||||
{
|
||||
var logger = new CapturingLogger<DeployedSecretsMirror>();
|
||||
var (mirror, storage, _, resolver, _) = await CreateAsync(logger);
|
||||
resolver.ThrowFor("a");
|
||||
|
||||
await storage.UpsertDeployedSecretsAsync([Art("a", 1, kek: "kek-1")], "d1");
|
||||
await mirror.ReconcileAsync(CancellationToken.None); // Error (rev 1)
|
||||
await mirror.ReconcileAsync(CancellationToken.None); // Debug (rev 1, repeat)
|
||||
|
||||
// Central ships a new revision of the same (still-failing) secret.
|
||||
await storage.UpsertDeployedSecretsAsync([Art("a", 2, kek: "kek-1")], "d2");
|
||||
await mirror.ReconcileAsync(CancellationToken.None); // Error again (rev 2 is new)
|
||||
|
||||
var levels = logger.Entries
|
||||
.Where(e => e.Message.Contains("did not decrypt"))
|
||||
.Select(e => e.Level)
|
||||
.ToList();
|
||||
Assert.Equal([LogLevel.Error, LogLevel.Debug, LogLevel.Error], levels);
|
||||
}
|
||||
|
||||
// ── Cancellation + concurrency (ruling 6) ──
|
||||
|
||||
[Fact]
|
||||
public async Task ReconcileAsync_PreCancelledToken_ThrowsOperationCanceled()
|
||||
{
|
||||
var (mirror, _, _, _, _) = await CreateAsync();
|
||||
using var cts = new CancellationTokenSource();
|
||||
cts.Cancel();
|
||||
|
||||
await Assert.ThrowsAnyAsync<OperationCanceledException>(
|
||||
() => mirror.ReconcileAsync(cts.Token));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ReconcileAsync_ConcurrentCallers_BothCompleteWithoutDoubleApply()
|
||||
{
|
||||
var (mirror, storage, store, _, _) = await CreateAsync();
|
||||
await storage.UpsertDeployedSecretsAsync([Art("a", 1)], "d1");
|
||||
|
||||
var t1 = mirror.ReconcileAsync(CancellationToken.None);
|
||||
var t2 = mirror.ReconcileAsync(CancellationToken.None);
|
||||
var results = await Task.WhenAll(t1, t2);
|
||||
|
||||
// The gate serializes the two passes: whichever runs first applies it, the other
|
||||
// sees it already ledgered at the shipped revision and applies nothing.
|
||||
Assert.Equal(1, results.Sum(r => r.Applied.Count));
|
||||
Assert.True(store.Contains("a"));
|
||||
|
||||
var mirrored = await storage.GetMirroredSecretsAsync();
|
||||
Assert.Equal(1, mirrored.Single(m => m.Name == "a").Revision);
|
||||
}
|
||||
|
||||
// ── Dispose (ruling 6) ──
|
||||
|
||||
[Fact]
|
||||
public void Dispose_DoesNotThrow_AndIsIdempotent()
|
||||
{
|
||||
var storage = new SiteStorageService(_localDb.Db, NullLogger<SiteStorageService>.Instance);
|
||||
var mirror = new DeployedSecretsMirror(
|
||||
storage,
|
||||
new FakeSecretStore(),
|
||||
new FakeSecretResolver(),
|
||||
invalidator: null,
|
||||
Options.Create(new SiteRuntimeOptions()),
|
||||
NullLogger<DeployedSecretsMirror>.Instance,
|
||||
TimeProvider.System);
|
||||
|
||||
mirror.Dispose();
|
||||
mirror.Dispose();
|
||||
}
|
||||
|
||||
// ── Hosted loop ──
|
||||
|
||||
[Fact]
|
||||
public async Task ExecuteAsync_RunsOnceAtStart_AndAgainEveryInterval()
|
||||
{
|
||||
var (mirror, storage, store, resolver, _) = await CreateAsync(
|
||||
interval: TimeSpan.FromMilliseconds(30));
|
||||
await storage.UpsertDeployedSecretsAsync([Art("a", 1)], "d1");
|
||||
|
||||
await mirror.StartAsync(CancellationToken.None);
|
||||
try
|
||||
{
|
||||
await WaitUntilAsync(() => store.Contains("a"));
|
||||
Assert.True(store.Contains("a")); // ran once immediately at start
|
||||
|
||||
// Prove the loop keeps ticking (not just the immediate start-of-day pass) by
|
||||
// observing further resolver invocations over time.
|
||||
var before = resolver.CallCount;
|
||||
await WaitUntilAsync(() => resolver.CallCount > before + 1, timeoutMs: 2000);
|
||||
Assert.True(resolver.CallCount > before + 1);
|
||||
}
|
||||
finally
|
||||
{
|
||||
await mirror.StopAsync(CancellationToken.None);
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ExecuteAsync_PerRowFailure_IsIsolated_AndLoopRecoversOnceCleared()
|
||||
{
|
||||
var logger = new CapturingLogger<DeployedSecretsMirror>();
|
||||
var (mirror, storage, store, _, _) = await CreateAsync(
|
||||
logger, interval: TimeSpan.FromMilliseconds(30));
|
||||
await storage.UpsertDeployedSecretsAsync([Art("a", 1)], "d1");
|
||||
store.ThrowOnApply = true;
|
||||
|
||||
await mirror.StartAsync(CancellationToken.None);
|
||||
try
|
||||
{
|
||||
// A per-row store fault is isolated (ruling 2): it never escapes ReconcileAsync,
|
||||
// so it shows up as a row-level Error log, not the pass-level Warning.
|
||||
await WaitUntilAsync(() => logger.Entries.Any(
|
||||
e => e.Level == LogLevel.Error && e.Message.Contains("failed to process")));
|
||||
|
||||
// The loop must still be alive after the failure — force success and confirm a
|
||||
// later pass applies the secret.
|
||||
store.ThrowOnApply = false;
|
||||
await WaitUntilAsync(() => store.Contains("a"), timeoutMs: 3000);
|
||||
Assert.True(store.Contains("a"));
|
||||
}
|
||||
finally
|
||||
{
|
||||
await mirror.StopAsync(CancellationToken.None);
|
||||
}
|
||||
}
|
||||
|
||||
// ── Harness ──
|
||||
|
||||
private async Task<(
|
||||
DeployedSecretsMirror Mirror,
|
||||
SiteStorageService Storage,
|
||||
FakeSecretStore Store,
|
||||
FakeSecretResolver Resolver,
|
||||
FakeSecretCacheInvalidator Invalidator)> CreateAsync(
|
||||
CapturingLogger<DeployedSecretsMirror>? logger = null,
|
||||
TimeSpan? interval = null,
|
||||
TimeProvider? time = null)
|
||||
{
|
||||
var storage = new SiteStorageService(_localDb.Db, NullLogger<SiteStorageService>.Instance);
|
||||
await storage.InitializeAsync();
|
||||
|
||||
var store = new FakeSecretStore();
|
||||
var resolver = new FakeSecretResolver();
|
||||
var invalidator = new FakeSecretCacheInvalidator();
|
||||
var options = Options.Create(new SiteRuntimeOptions
|
||||
{
|
||||
DeployedSecretsMirrorInterval = interval ?? TimeSpan.FromSeconds(15),
|
||||
});
|
||||
|
||||
var mirror = new DeployedSecretsMirror(
|
||||
storage,
|
||||
store,
|
||||
resolver,
|
||||
invalidator,
|
||||
options,
|
||||
(ILogger<DeployedSecretsMirror>?)logger ?? NullLogger<DeployedSecretsMirror>.Instance,
|
||||
time ?? TimeProvider.System);
|
||||
|
||||
return (mirror, storage, store, resolver, invalidator);
|
||||
}
|
||||
|
||||
private static SecretArtifact Art(
|
||||
string name, long revision, string kek = "kek-1", string contentType = "Text", byte[]? ciphertext = null) => new(
|
||||
Name: name,
|
||||
ContentType: contentType,
|
||||
Ciphertext: ciphertext ?? [1, 2, 3, 4, 5],
|
||||
Nonce: [10, 11, 12],
|
||||
Tag: [20, 21, 22, 23],
|
||||
WrappedDek: [30, 31, 32, 33, 34, 35],
|
||||
WrapNonce: [40, 41, 42],
|
||||
WrapTag: [50, 51, 52, 53],
|
||||
KekId: kek,
|
||||
Revision: revision,
|
||||
CreatedUtc: new DateTimeOffset(2026, 9, 28, 0, 0, 0, TimeSpan.Zero),
|
||||
UpdatedUtc: new DateTimeOffset(2026, 9, 28, 1, 0, 0, TimeSpan.Zero),
|
||||
Description: "test secret");
|
||||
|
||||
private static async Task WaitUntilAsync(Func<bool> condition, int timeoutMs = 5000)
|
||||
{
|
||||
var elapsed = 0;
|
||||
const int step = 20;
|
||||
while (!condition() && elapsed < timeoutMs)
|
||||
{
|
||||
await Task.Delay(step);
|
||||
elapsed += step;
|
||||
}
|
||||
Assert.True(condition(), "condition not met within timeout");
|
||||
}
|
||||
|
||||
/// <summary>Thread-safe capturing logger so tests can assert on emitted log levels/messages, mirroring SiteReconciliationActorTests.</summary>
|
||||
private sealed class CapturingLogger<T> : ILogger<T>
|
||||
{
|
||||
public ConcurrentQueue<(LogLevel Level, string Message)> Entries { get; } = new();
|
||||
|
||||
IDisposable? ILogger.BeginScope<TState>(TState state) => null;
|
||||
public bool IsEnabled(LogLevel logLevel) => true;
|
||||
|
||||
public void Log<TState>(
|
||||
LogLevel logLevel, EventId eventId, TState state, Exception? exception,
|
||||
Func<TState, Exception?, string> formatter)
|
||||
=> Entries.Enqueue((logLevel, formatter(state, exception)));
|
||||
}
|
||||
|
||||
/// <summary>Minimal controllable <see cref="TimeProvider"/> — mirrors the pattern used by <c>ApiMethodCacheTests</c>.</summary>
|
||||
private sealed class FakeTimeProvider : TimeProvider
|
||||
{
|
||||
public DateTimeOffset Now { get; set; }
|
||||
public FakeTimeProvider(DateTimeOffset start) => Now = start;
|
||||
public override DateTimeOffset GetUtcNow() => Now;
|
||||
}
|
||||
|
||||
/// <summary>Records every <see cref="ISecretCacheInvalidator.Invalidate"/> call by secret name.</summary>
|
||||
private sealed class FakeSecretCacheInvalidator : ISecretCacheInvalidator
|
||||
{
|
||||
public List<string> Invalidated { get; } = new();
|
||||
public void Invalidate(SecretName name) => Invalidated.Add(name.Value);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// In-memory <see cref="ISecretStore"/> honouring the REAL <c>SqliteSecretStore</c>
|
||||
/// semantics (fix round 1, review Important #1): <see cref="ApplyReplicatedAsync"/> is
|
||||
/// strict last-writer-wins via <see cref="SecretLastWriterWins.IsNewer"/> and applies
|
||||
/// verbatim (no revision bump); <see cref="DeleteAsync"/> tombstones in place (bumps
|
||||
/// revision, stamps <see cref="Now"/>) rather than hard-deleting.
|
||||
/// </summary>
|
||||
private sealed class FakeSecretStore : ISecretStore
|
||||
{
|
||||
private readonly Dictionary<string, StoredSecret> _rows = new(StringComparer.Ordinal);
|
||||
|
||||
/// <summary>
|
||||
/// Controllable "now" used to stamp a <see cref="DeleteAsync"/> tombstone —
|
||||
/// deterministic, not wall-clock, so tests can control ordering against a shipped
|
||||
/// artifact's fixed <c>UpdatedUtc</c>.
|
||||
/// </summary>
|
||||
public DateTimeOffset Now { get; set; } = new DateTimeOffset(2026, 9, 28, 12, 0, 0, TimeSpan.Zero);
|
||||
|
||||
/// <summary>Set to make the next <see cref="ApplyReplicatedAsync"/> call throw, simulating a store fault.</summary>
|
||||
public bool ThrowOnApply { get; set; }
|
||||
|
||||
/// <summary>Every row currently held, including tombstones, for direct inspection.</summary>
|
||||
public IReadOnlyDictionary<string, StoredSecret> Rows => _rows;
|
||||
|
||||
/// <summary>Seeds a LIVE row directly (bypassing ApplyReplicatedAsync/LWW) — an out-of-band secret.</summary>
|
||||
public void Seed(string name, DateTimeOffset updatedUtc, long revision = 1, byte[]? ciphertext = null) =>
|
||||
_rows[name] = new StoredSecret
|
||||
{
|
||||
Name = new SecretName(name),
|
||||
ContentType = SecretContentType.Text,
|
||||
Ciphertext = ciphertext ?? [9],
|
||||
Nonce = [9],
|
||||
Tag = [9],
|
||||
WrappedDek = [9],
|
||||
WrapNonce = [9],
|
||||
WrapTag = [9],
|
||||
KekId = "kek-out-of-band",
|
||||
Revision = revision,
|
||||
CreatedUtc = updatedUtc,
|
||||
UpdatedUtc = updatedUtc,
|
||||
};
|
||||
|
||||
/// <summary>True if the row is present and NOT tombstoned.</summary>
|
||||
public bool Contains(string name) => _rows.TryGetValue(name, out var row) && !row.IsDeleted;
|
||||
|
||||
/// <summary>True if the row is present as a tombstone.</summary>
|
||||
public bool IsTombstoned(string name) => _rows.TryGetValue(name, out var row) && row.IsDeleted;
|
||||
|
||||
public Task<StoredSecret?> GetAsync(SecretName name, CancellationToken ct) =>
|
||||
Task.FromResult(_rows.TryGetValue(name.Value, out var row) ? row : null);
|
||||
|
||||
public Task UpsertAsync(StoredSecret row, CancellationToken ct)
|
||||
{
|
||||
_rows[row.Name.Value] = row;
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
/// <summary>Tombstones (does not hard-delete) — matches <c>SqliteSecretStore.DeleteAsync</c>.</summary>
|
||||
public Task<bool> DeleteAsync(SecretName name, string? actor, CancellationToken ct)
|
||||
{
|
||||
if (!_rows.TryGetValue(name.Value, out var existing) || existing.IsDeleted)
|
||||
{
|
||||
return Task.FromResult(false);
|
||||
}
|
||||
|
||||
_rows[name.Value] = existing with
|
||||
{
|
||||
IsDeleted = true,
|
||||
DeletedUtc = Now,
|
||||
UpdatedUtc = Now,
|
||||
Revision = existing.Revision + 1,
|
||||
UpdatedBy = actor,
|
||||
};
|
||||
return Task.FromResult(true);
|
||||
}
|
||||
|
||||
public Task<IReadOnlyList<SecretMetadata>> ListAsync(bool includeDeleted, CancellationToken ct) =>
|
||||
Task.FromResult<IReadOnlyList<SecretMetadata>>([]);
|
||||
|
||||
public Task<IReadOnlyList<SecretManifestEntry>> GetManifestAsync(CancellationToken ct) =>
|
||||
Task.FromResult<IReadOnlyList<SecretManifestEntry>>([]);
|
||||
|
||||
/// <summary>Strict LWW on (UpdatedUtc, Revision) via <see cref="SecretLastWriterWins.IsNewer"/>; applied verbatim — matches <c>SqliteSecretStore</c>.</summary>
|
||||
public Task ApplyReplicatedAsync(StoredSecret row, CancellationToken ct)
|
||||
{
|
||||
if (ThrowOnApply)
|
||||
{
|
||||
throw new InvalidOperationException("simulated store fault");
|
||||
}
|
||||
|
||||
if (_rows.TryGetValue(row.Name.Value, out var existing)
|
||||
&& !SecretLastWriterWins.IsNewer(row.UpdatedUtc, row.Revision, existing.UpdatedUtc, existing.Revision))
|
||||
{
|
||||
return Task.CompletedTask; // no-op, matches real semantics
|
||||
}
|
||||
|
||||
_rows[row.Name.Value] = row;
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
public Task<bool> ApplyRewrapAsync(
|
||||
StoredSecret rewrappedRow, byte[] expectedCurrentWrappedDek, CancellationToken ct) =>
|
||||
Task.FromResult(false);
|
||||
}
|
||||
|
||||
/// <summary>In-memory <see cref="ISecretResolver"/> whose per-name resolution outcome tests control.</summary>
|
||||
private sealed class FakeSecretResolver : ISecretResolver
|
||||
{
|
||||
private readonly HashSet<string> _throwing = new(StringComparer.Ordinal);
|
||||
private readonly HashSet<string> _nullReturning = new(StringComparer.Ordinal);
|
||||
|
||||
public int CallCount;
|
||||
|
||||
/// <summary>When set, a name that is absent or tombstoned in this store resolves to null, like the real resolver.</summary>
|
||||
public FakeSecretStore? BackingStore { get; set; }
|
||||
|
||||
/// <summary>Row absent/tombstoned path — <c>DefaultSecretResolver.GetAsync</c> returns null.</summary>
|
||||
public void ReturnNullFor(string name) => _nullReturning.Add(name);
|
||||
|
||||
/// <summary>Decryption/integrity failure path — <c>DefaultSecretResolver.GetAsync</c> rethrows <c>SecretDecryptionException</c>.</summary>
|
||||
public void ThrowFor(string name) => _throwing.Add(name);
|
||||
|
||||
public void ClearFailures()
|
||||
{
|
||||
_nullReturning.Clear();
|
||||
_throwing.Clear();
|
||||
}
|
||||
|
||||
public Task<string?> GetAsync(SecretName name, CancellationToken ct)
|
||||
{
|
||||
Interlocked.Increment(ref CallCount);
|
||||
if (_throwing.Contains(name.Value))
|
||||
{
|
||||
throw new InvalidOperationException("simulated decrypt failure");
|
||||
}
|
||||
|
||||
if (BackingStore is not null && !BackingStore.Contains(name.Value))
|
||||
{
|
||||
return Task.FromResult<string?>(null);
|
||||
}
|
||||
|
||||
return Task.FromResult(_nullReturning.Contains(name.Value) ? null : "decrypted-value");
|
||||
}
|
||||
}
|
||||
}
|
||||
+9
@@ -15,6 +15,15 @@
|
||||
<PackageReference Include="FluentAssertions" />
|
||||
<PackageReference Include="Microsoft.NET.Test.Sdk" />
|
||||
<PackageReference Include="Moq" />
|
||||
<!--
|
||||
Task 9 fix round 2: DeployedSecretsMirrorRealStackTests runs against the REAL
|
||||
ZB.MOM.WW.Secrets stack (SqliteSecretStore + AesGcmEnvelopeCipher + DefaultSecretResolver)
|
||||
rather than the in-memory fakes DeployedSecretsMirrorTests uses, to pin the
|
||||
tombstone-recovery re-stamp's "does not break decryption" property against the real
|
||||
library instead of by source inspection alone. Test-only: SiteRuntime itself depends only
|
||||
on ZB.MOM.WW.Secrets.Abstractions.
|
||||
-->
|
||||
<PackageReference Include="ZB.MOM.WW.Secrets" />
|
||||
<PackageReference Include="xunit" />
|
||||
<PackageReference Include="xunit.runner.visualstudio" />
|
||||
</ItemGroup>
|
||||
|
||||
Reference in New Issue
Block a user