Compare commits
7 Commits
23a139a46c
...
2650f79472
| Author | SHA1 | Date | |
|---|---|---|---|
| 2650f79472 | |||
| 4e6409184c | |||
| f5756affef | |||
| 0289d51137 | |||
| 253939cd5d | |||
| 29049c4ef6 | |||
| ef03a0dca7 |
@@ -90,7 +90,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
|
||||
- DCL write failures returned synchronously to calling script.
|
||||
- Tag path resolution retried periodically for devices still booting.
|
||||
- Static attribute writes persisted to local SQLite (survive restart/failover, reset on redeployment).
|
||||
- **Consolidated site database (LocalDb Phase 1 + 2, complete 2026-07-20).** Eleven tables now live in ONE `ZB.MOM.WW.LocalDb`-managed SQLite file (the `SiteLocalDbSetup.ReplicatedTables` list — the 11th, `deployed_secrets`, was added 2026-09-28 by the site-secrets-via-artifact-deploy change, see *Templates & Deployment* below), configured by the **required** `LocalDb:Path` (`/app/data/site-localdb.db` on the rig; validated with `ValidateOnStart`, so a site config missing it fails to boot). Consequences worth knowing:
|
||||
- **Consolidated site database (LocalDb Phase 1 + 2, complete 2026-07-20).** Twelve tables now live in ONE `ZB.MOM.WW.LocalDb`-managed SQLite file (the `SiteLocalDbSetup.ReplicatedTables` list — the 11th, `deployed_secrets`, was added 2026-09-28 by the site-secrets-via-artifact-deploy change, see *Templates & Deployment* below; the 12th, `secret_mirror_status` — one row per node carrying that node's deployed-secrets mirror failures so the standby's reach central on the active node's health report — by review finding M1 the same day), configured by the **required** `LocalDb:Path` (`/app/data/site-localdb.db` on the rig; validated with `ValidateOnStart`, so a site config missing it fails to boot). Consequences worth knowing:
|
||||
- `site_events.id` changed from autoincrement INTEGER to an application-minted **GUID**. Last-writer-wins keys on the primary key, so two nodes independently minting `id=1,2,3…` would destroy each other's events rather than merge them. The event-log read path uses a composite `(timestamp, id)` keyset cursor with an **opaque string** continuation token; `EventLogEntry.Id` and both `ContinuationToken`s are `string`/`string?` on the site↔central Akka DTOs.
|
||||
- `ScadaBridge:OperationTracking:ConnectionString`, `ScadaBridge:SiteEventLog:DatabasePath` and — as of Phase 2 — `ScadaBridge:StoreAndForward:SqliteDbPath` + `ScadaBridge:Database:SiteDbPath` are all **migration-only** — nothing reads them but `SiteLocalDbLegacyMigrator`, which copies a pre-Phase-1 file in once (deterministic `mig-{NodeName}-{legacyId}` event ids, `INSERT OR IGNORE`, runs AFTER `RegisterReplicated` so migrated rows replicate) and renames it `.migrated`. Delete the keys once a node has migrated.
|
||||
- This incidentally fixes a data-loss bug: both legacy databases defaulted to CWD-relative paths **outside** the mounted volume and were discarded on every container recreate.
|
||||
@@ -101,10 +101,10 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
|
||||
- **`notification_lists` and `smtp_configurations` are created but deliberately NOT registered.** They are permanently empty on a site (no writer since 2026-07-10, the migrator skips them, the active-node purge keeps them empty), and registering them would open a standing replication channel whose only historical payload was plaintext SMTP passwords. Pinned by a security-named test, and verified live: those two tables have **no CDC triggers** on either rig node.
|
||||
- **Operational constraints (read before upgrading a site pair):** stop and start both nodes TOGETHER — rolling one at a time is no longer supported, since the legacy `SfBufferSnapshot` compatibility handler went with the replicator. And a node offline longer than `LocalDb:Replication:TombstoneRetention` (default 7 days) can resurrect deleted rows on rejoin. See `docs/deployment/topology-guide.md`.
|
||||
- **Batching is by BYTE BUDGET as of LocalDb 0.2.0** — `LocalDb:Replication:MaxBatchBytes` (default **2 MB**, sized under the 4 MB gRPC cap) bounds a delta/snapshot message by summed serialized size via a per-message split in `SyncSession.PumpLoopAsync`, with `MaxBatchSize` demoted to a secondary row cap; a single row over budget is sent alone rather than stalling the stream. **The rig's old `MaxBatchSize = 16` pin (a hand-computed byte-budget proxy: ~70 KB worst-case `config_json` x the 500 default is ~35 MB) is retired, but `MaxBatchSize` is NOT fully redundant with `MaxBatchBytes`** — it also bounds the separate DB READ page in `OplogStore.ReadBatchAboveAsync`/`SnapshotStreamer`, which materializes the whole page into memory *before* the byte-budget split runs, so an unset (500-default) `MaxBatchSize` still lets a reconnect drain transiently allocate ~35 MB per read even though every wire message stays under `MaxBatchBytes` (arch-review adversarial finding F2). Both site-a nodes on `docker/` therefore pin an explicit `"MaxBatchSize": 64` to bound that transient allocation, while `MaxBatchBytes` stays unset (its 2 MB default) to bound the wire message; site-b/site-c stay unreplicated so the key doesn't apply there.
|
||||
- **CDC registration is conditional, and both directions self-heal at boot** (arch-review WP1.3 + WP3.3, `Host/SiteLocalDbSetup.cs`). Capture triggers are installed only when this node has replication configured — `PeerAddress` **or** `ApiKey`, an OR because only the dialling half sets `PeerAddress` while the passive half carries the key alone. An unreplicated node calls **`DeregisterReplicated`** on all eleven tables at boot, dropping triggers an earlier build left behind and pruning their oplog/row-version rows (idempotent; logs once at Information when something was actually cleaned). A replicating node registers with **`baselineExistingRows: true`**, which seeds `__localdb_row_version` for pre-existing rows at the LWW floor (HLC `0`, this node's id) and flags a snapshot resync — so turning replication ON for a site that has been running without it now converges on the rows already in the file instead of only on writes made after the restart. Deregistration must be symmetric (the handshake compares registered-table digests fail-closed), so replication is a both-nodes-together change in either direction. LocalDb 0.2.0 also makes backlog depth O(1) and drops the unused `__localdb_oplog_hlc` index; the on-disk bookkeeping schema goes to v2, upgraded in place on open, with no wire change (0.1.x peers still sync).
|
||||
- **CDC registration is conditional, and both directions self-heal at boot** (arch-review WP1.3 + WP3.3, `Host/SiteLocalDbSetup.cs`). Capture triggers are installed only when this node has replication configured — `PeerAddress` **or** `ApiKey`, an OR because only the dialling half sets `PeerAddress` while the passive half carries the key alone. An unreplicated node calls **`DeregisterReplicated`** on all twelve tables at boot, dropping triggers an earlier build left behind and pruning their oplog/row-version rows (idempotent; logs once at Information when something was actually cleaned). A replicating node registers with **`baselineExistingRows: true`**, which seeds `__localdb_row_version` for pre-existing rows at the LWW floor (HLC `0`, this node's id) and flags a snapshot resync — so turning replication ON for a site that has been running without it now converges on the rows already in the file instead of only on writes made after the restart. Deregistration must be symmetric (the handshake compares registered-table digests fail-closed), so replication is a both-nodes-together change in either direction. LocalDb 0.2.0 also makes backlog depth O(1) and drops the unused `__localdb_oplog_hlc` index; the on-disk bookkeeping schema goes to v2, upgraded in place on open, with no wire change (0.1.x peers still sync).
|
||||
- **LocalDb 0.3.0 (2026-08-15) makes that snapshot flag PER-TABLE** — the residual 0.2.1 explicitly deferred. On-disk bookkeeping schema goes to **v3** (a new `__localdb_snapshot_state` table, upgraded in place on open); still wire-compatible, and `lib_schema_version` deliberately stays `1` because the handshake compares it fail-closed for *equality*. The extension is negotiated by **capability** (`Handshake.supports_partial_snapshot`, `SnapshotBegin.tables` where **empty == all tables**), so a peer that does not advertise it is served a full snapshot — a scoped one would make it advance `last_applied_remote_seq` past deltas it never received. For the same reason **a partial snapshot moves no watermark on either side**; the overlapping deltas are still pumped and absorbed by LWW. Snapshot debt now clears on the receiver's new `SnapshotAck` rather than on enqueue, so a crash mid-snapshot re-streams the same scope. **`SiteLocalDbSetup.cs` needed no restructuring** — it already calls `RegisterReplicated` once per table in a loop, which is exactly what per-table flagging keys off. The consequence to know: **adding an eleventh table to `ReplicatedTables` on an already-replicating site now snapshots that one table**, where through 0.2.x the same edit re-streamed all eleven in full in both directions. First-boot behaviour is unchanged (all ten seed at once ⇒ the flagged set is every registered table ⇒ an ordinary full snapshot), and so is upgrading the rig in place (the ten tables are already ledgered, nothing seeds, nothing is flagged). A future "register the whole list in one call" API would silently give the coarse behaviour back.
|
||||
- **LocalDb 0.3.2 (2026-08-19, ScadaBridge arch-review P4-1 + P4-3; wire-compatible, no schema change):** (1) the sync pump now **faults the live session when a cap-prune invalidates its position** (horizon check on the oplog's min seq before each read batch + on the batch head after it; throws the new public `LocalDbOplogGapException` through the 0.3.1 fault-teardown) — previously a slow-but-alive peer silently skipped the pruned gap and the pair diverged until the next reconnect; (2) **transport keepalives both sides** — client `SocketsHttpHandler` pings via new validated `LocalDb:Replication:KeepAlivePingDelay`/`KeepAlivePingTimeout` (defaults 60s/20s, `Always` policy), and ScadaBridge's site gRPC listener sets Kestrel h2 keepalive (`Host/SiteKestrelSetup.cs` — server-wide, also governs the metrics port); (3) the server's single-stream guard is now **newest-wins with an awaited handover** — a fresh authenticated `Sync` cancels a dead/wedged incumbent and *waits* for it to finish (`TakeoverTimeout` 10s) before starting, so a half-open link can no longer wedge the pair behind `AlreadyExists` while the zombie side reports `LocalDbReplicationConnected=true`.
|
||||
- **LocalDb 0.3.3 (2026-08-20, ScadaBridge arch-review P4-7; wire-compatible, no schema change — superseded by 0.3.4 below in the same bullet):** two latent `LwwApplier` hazards closed before any table could hit them. (1) **`RegisterReplicated` now REFUSES a table that is the parent of a foreign key with a row-mutating `ON DELETE` action (`CASCADE`/`SET NULL`/`SET DEFAULT`) from any other table** (`LocalDbRegistrationException` naming `child.column ON DELETE <action>`; `RESTRICT`/`NO ACTION` still allowed) — the applier's `INSERT OR REPLACE` runs with `foreign_keys=ON`, so the implicit delete fired those actions while CDC capture was suppressed, i.e. child rows changed on one node and never replicated. Registration-time only: SQLite can't add an FK to an existing table, so a later-created child is caught at the next boot's registration. None of ScadaBridge's eleven tables has an FK (verified on the live rig file: zero `REFERENCES` across 29 `CREATE TABLE`s) — **keep it that way; a cascading-FK parent can never be one of `ReplicatedTables`** (register row 105). (2) The per-entry apply (delete-or-upsert + row-version ledger write) is now **atomic under a `SAVEPOINT`** — a fault between the two statements used to leave a deleted row with a live ledger entry, which `SnapshotStreamer` warns-and-skips forever; poison-catch semantics unchanged (narrow `SqliteException`, per-entry dead-letter, batch continues). Cost: two extra in-memory statements per applied entry on a backlog drain. **Both are pinned here** (2026-08-20): `SiteLocalDbReplicatedTableFkGuardTests` asserts at build time that no `ReplicatedTables` entry has an inbound row-mutating FK (register row 105), and **pre-2026-08-20 site files must be VACUUMed once offline** to get `auto_vacuum=INCREMENTAL` — `docker/vacuum-site-localdb.sh` (pair stopped together; done on all three rig pairs; topology guide → *one-off offline VACUUM*; row 104). **LocalDb 0.3.4 (2026-08-20, register row 106; wire-compatible, sender-side only):** a converged pair whose sender oplog was ack-pruned to empty used to stamp snapshots `as_of_seq 0`, so a peer with a zero watermark (a rebuilt node — the rig's site-a-a since 2026-08-15) received a FULL snapshot on every reconnect until the sender wrote again (127 k rows per restart, observed). The as-of now floors at `last_acked_seq`; proven live — second restart after the pin exchanged zero snapshots. Lesson for rebuilding a node from its peer: the first reconnect snapshots, the second must not.
|
||||
- **LocalDb 0.3.3 (2026-08-20, ScadaBridge arch-review P4-7; wire-compatible, no schema change — superseded by 0.3.4 below in the same bullet):** two latent `LwwApplier` hazards closed before any table could hit them. (1) **`RegisterReplicated` now REFUSES a table that is the parent of a foreign key with a row-mutating `ON DELETE` action (`CASCADE`/`SET NULL`/`SET DEFAULT`) from any other table** (`LocalDbRegistrationException` naming `child.column ON DELETE <action>`; `RESTRICT`/`NO ACTION` still allowed) — the applier's `INSERT OR REPLACE` runs with `foreign_keys=ON`, so the implicit delete fired those actions while CDC capture was suppressed, i.e. child rows changed on one node and never replicated. Registration-time only: SQLite can't add an FK to an existing table, so a later-created child is caught at the next boot's registration. None of ScadaBridge's twelve replicated tables has an FK (verified on the live rig file: zero `REFERENCES` across 29 `CREATE TABLE`s) — **keep it that way; a cascading-FK parent can never be one of `ReplicatedTables`** (register row 105). (2) The per-entry apply (delete-or-upsert + row-version ledger write) is now **atomic under a `SAVEPOINT`** — a fault between the two statements used to leave a deleted row with a live ledger entry, which `SnapshotStreamer` warns-and-skips forever; poison-catch semantics unchanged (narrow `SqliteException`, per-entry dead-letter, batch continues). Cost: two extra in-memory statements per applied entry on a backlog drain. **Both are pinned here** (2026-08-20): `SiteLocalDbReplicatedTableFkGuardTests` asserts at build time that no `ReplicatedTables` entry has an inbound row-mutating FK (register row 105), and **pre-2026-08-20 site files must be VACUUMed once offline** to get `auto_vacuum=INCREMENTAL` — `docker/vacuum-site-localdb.sh` (pair stopped together; done on all three rig pairs; topology guide → *one-off offline VACUUM*; row 104). **LocalDb 0.3.4 (2026-08-20, register row 106; wire-compatible, sender-side only):** a converged pair whose sender oplog was ack-pruned to empty used to stamp snapshots `as_of_seq 0`, so a peer with a zero watermark (a rebuilt node — the rig's site-a-a since 2026-08-15) received a FULL snapshot on every reconnect until the sender wrote again (127 k rows per restart, observed). The as-of now floors at `last_acked_seq`; proven live — second restart after the pin exchanged zero snapshots. Lesson for rebuilding a node from its peer: the first reconnect snapshots, the second must not.
|
||||
- **LocalDb 0.2.1** (two adversarial-review findings against 0.2.0, wire-compatible, no schema change) fixes: (1) `DeregisterReplicated` was deleting the HLC clock's only durable crash anchor along with the table rows — an ungraceful exit (crash, SIGKILL, power loss) on a fully-deregistered file left nothing to recover from, so the reopened clock could restart from wall-clock time and re-issue stamps below HLCs peers already hold (silently discarded by LWW); the current clock is now flushed inside the deregistration transaction, before the prune. (2) The sync inbox was an **unbounded** channel, so inbound memory was a function of what the peer sends — a mutual full snapshot (both sides of a pair baseline unconditionally when replication is enabled late) buffered whole, in memory, on both nodes at once; it is now **bounded at 64 messages** with wait-mode backpressure (unrelated to, and not to be confused with, the rig's separate `MaxBatchSize: 64` DB-read-page pin above), which required moving the snapshot to run alongside the receive loop (not inside the pre-loop handshake) so the inbox is drained for the whole session and can't deadlock itself.
|
||||
- All timestamps are UTC throughout the system.
|
||||
- Inter-cluster communication uses **three** transports, not two — **all cross-cluster command/control and data now rides gRPC** after the ClusterClient→gRPC migration's Phase 4 (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`) deleted Akka `ClusterClient`/`ClusterClientReceptionist`: (1) **gRPC command/control** — site→central over the central-hosted `CentralControlService` (`GrpcCentralTransport`, sticky central-a→central-b channel pair; deployments/notifications/health/heartbeat/audit-ingest/reconcile), and central→site over the site-hosted `SiteCommandService` (`GrpcSiteTransport`, per-site NodeA→NodeB channel pair; the 28 lifecycle/OPC-UA/query/parked/route/failover commands); (2) **gRPC** server-streaming for real-time data (attribute values, alarm states, `SiteStreamService`); and (3) **plain token-gated HTTP** for the deployment config itself — notify-and-fetch, the site pulls the config from `DeploymentConfigEndpoints` (`ManagementService/DeploymentConfigEndpoints.cs`) with an `X-Deployment-Token` header, `AllowAnonymous` with the per-deployment token as the entire security boundary. The gRPC boundary is per-site PSK-authenticated (`ControlPlaneAuthInterceptor`, unchanged). There is **no receptionist registration** — discovery is by dialling configured endpoints; central builds one `SitePairChannelProvider` per site (addresses from `Site.GrpcNodeAAddress`/`GrpcNodeBAddress`, refreshed from the DB every 60s and on admin changes), sites dial `ScadaBridge:Communication:CentralGrpcEndpoints` (both central nodes, h2c on `CentralGrpcPort` 8083, **NOT** via Traefik). **Discovery is asymmetric by design:** central discovers site gRPC addresses from the *database* (refreshable at runtime), sites discover central from *appsettings* (`CentralGrpcEndpoints`, static — restart required; `StartupValidator` requires a Site node to list at least one). `Akka.Cluster.Tools` stays for ClusterSingleton; only the ClusterClient part is gone. **Central never buffers for an unreachable site** — the send fails with the caller's Ask/deadline timing out; a `ConnectionStateChanged` mechanism built for this was deleted as dead code.
|
||||
@@ -141,7 +141,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
|
||||
- Per-instance operation lock covers all mutating commands (deploy, disable, enable, delete).
|
||||
- **Bulk site deployment (`DeploymentService.DeploySiteAsync`, arch-review WP2.5)** redeploys every out-of-date instance at a site through one flatten session (a template chain shared by N instances is walked once) in three phases — serial Prepare, bounded-parallel Send (`SiteDeploymentMaxParallelism`, default 4), serial Finalize — still **not all-or-nothing**: any one instance's failure doesn't block the rest. **Disabled instances are SKIPPED by bulk deploy as of 2026-08-19 (arch-review P7-2)** — they are reported in the summary's additive `SkippedInstances` list, never flattened/sent/re-enabled (single-instance deploy remains the explicit operator path that deploys+enables a Disabled instance); **Up-to-date instances are ALSO skipped as of 2026-08-20 (arch-review P7-6)** — a redeploy is not idempotent at the site (it resets static overrides and clears native alarm state), so `DeploySiteAsync` pre-filters through `IStaleInstanceProbe` (trustworthy since P7-4 fixed its memo timing) and reports unchanged instances in `SkippedInstances` with reason `up-to-date`; the filter is conservative (every uncertainty deploys: missing snapshot, `ShouldQuerySiteBeforeRedeploy`, null/faulting probe). Additive `MgmtDeploySiteCommand.Force` / CLI `deploy site --force` restores the unconditional push for what central cannot see (restored volume, rebuilt node) — **Disabled still wins under `--force`**. Also P7-5: the three phases share one scoped DbContext and a failed `SaveChanges` leaves the entity tracked, so one permanent failure used to cascade to every later instance and strand records `InProgress`; `MarkDeploymentFailedAsync` now detaches pending changes and retries the `Failed` write once on the clean tracker; Prepare's per-instance reconcile also runs under a short `SiteDeploymentReconcileTimeout` (5s) with a once-per-batch unreachable-site circuit-breaker, so a down site no longer costs 30s × N serial before the first send (P7-3). The `PendingDeployment` staging row (5-minute TTL) is written per-instance immediately before that instance's round-trip in the Send phase, not upfront in Prepare, so a large batch's tail instances don't have their TTL clock already half-spent by the time they're reached.
|
||||
- **Site artifact revision + fleet-wide pending-updates indicator (2026-09-14, `docs/plans/2026-09-14-pending-updates-design.md`).** `Site.ArtifactRevisionHash`/`ArtifactsDeployedAt` (migration `AddSiteArtifactRevision`) are stamped by `ArtifactDeploymentService` only on a site whose artifact deploy succeeds, so a redeployed fleet's artifacts are trackable per site the same way instance staleness already is. `PendingUpdatesEvaluator` combines that with `DeploymentService.GetInstanceStalenessAsync` (one shared `FlattenSession` for the whole fleet) into a `PendingUpdatesSnapshot`, held behind a process-wide singleton `IPendingUpdatesCache` (`ScadaBridge:DeploymentManager:PendingUpdatesCacheTtl`, default 60s) so N Blazor circuits cost one fleet flatten per TTL, not one each. Surfaces: the Sites page's per-site **Artifacts** badge (`Current`/`Out of date`/`Never deployed`) + instance-staleness line + header **Re-check**, and a rail-footer **pending updates** indicator (Deployer/Administrator only, 30s poll, site-scoped). **Upgrade consequence:** every site reads *Never deployed* until one artifact deploy runs post-upgrade — deliberate, not a bug. Also shipped: the rail nav now sits in its own scroll pane (`NavMenu`'s `.rail-scroll` + `site.css`) so a long nav can't push the footer/indicator off-screen; the `ZB.MOM.WW.Theme` kit itself is untouched (register row 123 proposes upstreaming it).
|
||||
- **Site secrets ride the artifact deploy; the gRPC secrets hub is retired in ScadaBridge (2026-09-28, `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`).** `Deploy Artifacts` now ships each site **only the secrets it references** — `SiteSecretsCollector` scans a site's data-connection JSON for `secret:<name>` (`SecretReferenceScanner`, protocol-agnostic, bootstrap names like `SB-GRPC-PSK-*` excluded) and resolves each against the central `ISecretStore`; a missing/tombstoned secret fails **that site's** deploy before anything is sent (fail-closed), ciphertext only ever crosses the wire, and secrets factor into `ArtifactRevisionHashService`'s staleness hash (a missing secret hashes as a distinguishable marker, so an unresolvable reference never reads *Current*; a site with no secret references hashes identically whether the field is null or empty). A pre-feature site (answers with no `SecretsApplied`) or a partial apply is also treated as a deploy failure and is never stamped; one `DeploySiteSecrets`/`SiteSecrets` audit row (`secrets` name + revision and `missing` names only, never ciphertext; no `removed` field — a removal-only deploy writes no central row) is written per site deploy that carried secrets or a missing-secret failure, on both the sweep and the per-site retry paths. Site-side: `deployed_secrets` is the 11th LocalDb-replicated table (ciphertext rows only, upsert-then-soft-remove); the per-node `DeployedSecretsMirror` (`SiteRuntime:DeployedSecretsMirrorInterval`, default 15s, also run synchronously right after an active-node apply) reconciles it into that node's local `ISecretStore`, ledgers what it mirrored in the node-local `mirrored_secrets` table so an out-of-band local secret sharing a deployed name is never silently touched (a mirror-owned name deleted or edited locally is healed back to the deployed value on the next pass, even at an unchanged revision), and trial-decrypts every pass to catch a KEK mismatch. Secrets are mirrored before the deploy's data connections reach the DCL; a rotation with unchanged connection config takes effect on the next (re)connect. **A site pair using deployed secrets must run LocalDb replication** — otherwise only the node active at deploy time holds them. Central runs one shared secret store, `Secrets:Store:Provider=SqlServer` (Central role only; a clustered central left on `Sqlite` gets a boot Warning) — the pre-host `${secret:}` expander reads the same store the running node serves. **ALL secrets replication is retired in ScadaBridge** — the pull-only gRPC hub and the SQL-Server replication mode both — `Secrets:Replication:Enabled` set to anything but empty/`false` now fails `StartupValidator` on either role with a message pointing at Deploy Artifacts / `Secrets:Store:Provider=SqlServer`.
|
||||
- **Site secrets ride the artifact deploy; the gRPC secrets hub is retired in ScadaBridge (2026-09-28, `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`).** `Deploy Artifacts` now ships each site **only the secrets it references** — `SiteSecretsCollector` scans a site's data-connection JSON for `secret:<name>` (`SecretReferenceScanner`, protocol-agnostic, bootstrap names like `SB-GRPC-PSK-*` excluded) and resolves each against the central `ISecretStore`; a missing/tombstoned secret fails **that site's** deploy before anything is sent (fail-closed), ciphertext only ever crosses the wire, and secrets factor into `ArtifactRevisionHashService`'s staleness hash (a missing secret hashes as a distinguishable marker, so an unresolvable reference never reads *Current*; a site with no secret references hashes identically whether the field is null or empty). A pre-feature site (answers with no `SecretsApplied`) or a partial apply is also treated as a deploy failure and is never stamped; one `DeploySiteSecrets`/`SiteSecrets` audit row (`secrets` name + revision, `missing` and `removed` names, never ciphertext — `removed` comes from the site's `SecretsRemoved` reply, so a removal-only deploy is audited too) is written per site deploy that carried, lacked or retired secrets, on both the sweep and the per-site retry paths. A central secret-store read failure fails only the sites that reference a secret and leaves them *Out of date* in the pending-updates check — it no longer aborts the sweep or the fleet-wide check — and both artifact-deploy CLI commands exit 1 when any site failed. Site-side: `deployed_secrets` is the 11th LocalDb-replicated table (ciphertext rows only, upsert-then-soft-remove); the per-node `DeployedSecretsMirror` (`SiteRuntime:DeployedSecretsMirrorInterval`, default 15s, also run synchronously right after an active-node apply) reconciles it into that node's local `ISecretStore`, ledgers what it mirrored in the node-local `mirrored_secrets` table so an out-of-band local secret sharing a deployed name is never silently touched (a mirror-owned name deleted or edited locally is healed back to the deployed value on the next pass, even at an unchanged revision), and trial-decrypts every pass to catch a KEK mismatch. Each node also records its current mirror failures (names + messages only) in the 12th replicated table `secret_mirror_status` — written only when its failure set changes, plus once per start-up — and the active node forwards every node's row as `SiteHealthReport.SecretMirrorStatuses` (proto field 33), so a **standby** KEK mismatch shows as a *Secret mirror failing* badge on the Health dashboard instead of only in the standby's log (review finding M1). Secrets are mirrored before the deploy's data connections reach the DCL; a rotation with unchanged connection config takes effect on the next (re)connect. **A site pair using deployed secrets must run LocalDb replication** — otherwise only the node active at deploy time holds them. Central runs one shared secret store, `Secrets:Store:Provider=SqlServer` (Central role only; a clustered central left on `Sqlite` gets a boot Warning) — the pre-host `${secret:}` expander reads the same store the running node serves. **ALL secrets replication is retired in ScadaBridge** — the pull-only gRPC hub and the SQL-Server replication mode both — `Secrets:Replication:Enabled` set to anything but empty/`false` now fails `StartupValidator` on either role with a message pointing at Deploy Artifacts / `Secrets:Store:Provider=SqlServer`.
|
||||
- **Create Instance picks its template from the Templates folder tree** (2026-08-21): the flat `<select>` is gone. The Template field is a read-only folder-qualified path box (`TemplatePathFormatter`, "Folder / Sub / Name") + a **Choose…** button, both opening `TemplatePickerDialog` — an `IDialogService.ShowAsync` body wrapping the **same shared `TemplateFolderTree`** the Templates page renders, with its own search, expand/collapse-all, templates-only selection and double-click-to-pick. It opens on the current pick via the tree's additive **`RevealKeys`** parameter (ancestors of the named `f:`/`t:` keys start expanded when no `Filter` is active; an active filter's reveal set still wins; callers passing nothing are byte-identical). `InstanceCreate` gained `[SupplyParameterFromQuery] TemplateId`, which the Templates tree's new **Deployer-gated "New Instance…"** context-menu item navigates with; combined with `?siteId=` it populates the Connection Bindings card on first render. **The trap for anyone extending the picker:** in `TreeViewSelectionMode.Single` the inner `TreeView` reads the *singular* `SelectedKey`/`SelectedKeyChanged` pair, but `TemplateFolderTree` forwards only the plural (Checkbox-mode) `SelectedKeys` pair — so a Single-mode caller cannot get a row highlight through the shared tree and must own selection in its own `NodeContent`, as the picker does. Design: `docs/plans/2026-08-21-template-picker-design.md`.
|
||||
- **Create Instance binds module groups at creation** (2026-08-21): the create page's Connection Bindings card flattens template+site through the additive `IFlatteningPipeline.PreviewAsync` (an unsaved instance — no validation, no hash, nothing persisted, instance repo never touched) and reuses `BindingGroupBuilder`; `NodeBrowserDialog.SelectableNodeClasses` lets it pick **Object** nodes (default stays Variable-only for every existing caller). Create = `CreateInstanceAsync` → `SetConnectionBindingsAsync` (the existing upsert, no new server surface) → always navigate to Configure with `?created=1&bound=N|&bindError=…`, because the row exists whether or not the second call succeeded. Partial binding is legal: a group missing a connection or an Object is skipped and stays unbound. Design: `docs/plans/2026-08-21-create-instance-bindings-design.md`.
|
||||
- Site-side apply is all-or-nothing per instance.
|
||||
@@ -175,7 +175,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
|
||||
- One row per lifecycle event; cached calls produce 4+ rows per operation (`Submitted`, `Forwarded`, `Attempted`, `Delivered`/`Parked`/`Discarded`).
|
||||
- `ExecutionId` (`uniqueidentifier NULL`) is the universal per-run correlation value — every audit row emitted by one script execution / inbound request shares it; `CorrelationId` remains the per-operation lifecycle id (NULL for sync one-shots).
|
||||
- `ParentExecutionId` (`uniqueidentifier NULL`) is the cross-execution spawn pointer — every row of a spawned run carries the spawner's `ExecutionId`; bridges inbound API → routed-site-script, alarm-triggered on-trigger scripts, and nested `CallScript`/`CallShared` invocations; `IX_AuditLog_ParentExecution` backs the filter + the recursive execution-tree walk. **Tag-cascade (alarm leg) is populated, not just plumbed** — M5.4 T4 threaded the `parentExecutionId` parameter but every `AlarmActor.SpawnAlarmExecution` call site passed null, so alarm runs were silently always roots; the id now rides site-locally as `SetStaticAttributeCommand.SourceExecutionId` → `AttributeValueChanged.SourceExecutionId` → `SpawnAlarmExecution` (all additive/nullable, no wire/proto/schema change; `Expression` triggers capture the writer *with* the evaluated snapshot since the eval completes off-dispatcher). Sources are `ScriptRuntimeContext.SetAttribute` (the run's own `ExecutionId`) and `Route.To(...).SetAttributes(...)` (the inbound request's). **Still roots by design, not omission:** alarms fired by DCL data (external values have no spawning execution — including the device echo of a script write to a *data-sourced* attribute, so only **static** writes cascade), and `ScriptActor` value-change/conditional/expression/timer trigger runs (a timer tick has no spawner; a `WhileTrue`/interval run has no single identifiable write).
|
||||
- Site SQLite hot-path first, then gRPC telemetry to central; ingest is idempotent on `EventId`; periodic reconciliation pull as fallback when telemetry is lost. **The site audit DB is a separate, unreplicated SQLite file — NOT one of LocalDb's eleven consolidated tables** — at the **required** `AuditLog:SiteWriter:DatabasePath` (arch-review WP1.2, finding #2 High), `/app/data/auditlog.db` on the rig; `StartupValidator` refuses to boot a Site node without it. `DatabasePath` used to default to CWD-relative `auditlog.db`, which on the docker rig landed on the container's ephemeral overlay, not the mounted volume — the same data-loss pattern the pre-LocalDb legacy S&F/tracking databases hit (see the LocalDb bullet above), independently rediscovered here because this file sits outside LocalDb's management. The writer loop now also honors `FlushIntervalMs` (was validated but never read — one fsync'ing commit per event even at trickle rate) and sets `PRAGMA synchronous = NORMAL` (audit is best-effort by design, so the narrower power-loss window is an acceptable trade for far fewer fsyncs; WAL still guarantees no corruption).
|
||||
- Site SQLite hot-path first, then gRPC telemetry to central; ingest is idempotent on `EventId`; periodic reconciliation pull as fallback when telemetry is lost. **The site audit DB is a separate, unreplicated SQLite file — NOT one of LocalDb's twelve consolidated tables** — at the **required** `AuditLog:SiteWriter:DatabasePath` (arch-review WP1.2, finding #2 High), `/app/data/auditlog.db` on the rig; `StartupValidator` refuses to boot a Site node without it. `DatabasePath` used to default to CWD-relative `auditlog.db`, which on the docker rig landed on the container's ephemeral overlay, not the mounted volume — the same data-loss pattern the pre-LocalDb legacy S&F/tracking databases hit (see the LocalDb bullet above), independently rediscovered here because this file sits outside LocalDb's management. The writer loop now also honors `FlushIntervalMs` (was validated but never read — one fsync'ing commit per event even at trickle rate) and sets `PRAGMA synchronous = NORMAL` (audit is best-effort by design, so the narrower power-loss window is an acceptable trade for far fewer fsyncs; WAL still guarantees no corruption).
|
||||
- Cached operations: site emits a single additively-extended `CachedCallTelemetry` packet carrying both audit events and operational state; central writes `AuditLog` + `SiteCalls` in one transaction.
|
||||
- Payload cap 8 KB by default / 64 KB on error rows; auth headers redacted by default; SQL parameter values captured by default; per-target redaction opt-in. Inbound API: full verbatim capture up to `InboundMaxBytes` (default 1 MiB); request headers stored in `Extra.requestHeaders` (post-redaction); per-method `SkipBodyCapture` flag suppresses bodies while still recording headers + metadata; `AuditInboundCeilingHits` counter surfaced on health snapshot. (M5.3 T7)
|
||||
- Audit-write failure NEVER aborts the user-facing action — audit is best-effort, the action's own success/failure path is authoritative.
|
||||
|
||||
+111
-30
@@ -22,19 +22,24 @@ set -euo pipefail
|
||||
# Ciphertext is compared by SHA-256 only; secret VALUES are random throwaways held in shell
|
||||
# variables and handed to the browser through the environment — never printed, never on argv.
|
||||
#
|
||||
# Checks (each prints PASS/FAIL; the run ends "PASS n/8" and exits non-zero on any FAIL):
|
||||
# Checks (each prints PASS/FAIL; the run ends "PASS n/9" and exits non-zero on any FAIL):
|
||||
# 1 secret + secret-referencing MxGateway connection on site-a -> site-a Out of date, b/c Current
|
||||
# 2 deploy site-a -> success, secretsDeployed 1, site-a Current
|
||||
# 3 both site-a nodes hold the secret at the central revision + ciphertext (standby <= 30 s)
|
||||
# 4 rotate at central -> site-a Out of date -> deploy -> both nodes at the new revision
|
||||
# 5 drop the secret: reference -> deploy -> both nodes no longer hold it; an out-of-band secret
|
||||
# seeded on site-a-a through the library's own write path (the ZB.MOM.WW.Secrets `secret` CLI
|
||||
# run against that node's store) survives
|
||||
# 6 reference a non-existent secret -> site-a deploy fails closed with the named message,
|
||||
# site-a stays Out of date, nothing changed on either site-a node
|
||||
# 7 the same reference on site-b (deliberately mismatched dev KEK) -> deploy fails naming the KekId
|
||||
# 8 one DeploySiteSecrets audit row per secret-bearing deploy; no plaintext / base64 ciphertext
|
||||
# in any of the 8 nodes' logs
|
||||
# 5 standby master-key mismatch reaches central (review M1): site-a-b is recreated with a WRONG
|
||||
# KEK -> central's site-a health report shows node-b failing to decrypt the secret while
|
||||
# node-a is clean; site-a-b restored with the right KEK -> node-b's failures clear
|
||||
# 6 drop the secret: reference -> deploy -> both nodes no longer hold it and the reply names it
|
||||
# in secretsRemoved; an out-of-band secret seeded on site-a-a through the library's own write
|
||||
# path (the ZB.MOM.WW.Secrets `secret` CLI run against that node's store) survives
|
||||
# 7 reference a non-existent secret -> site-a deploy fails closed with the named message, the
|
||||
# CLI exits 1, site-a stays Out of date, nothing changed on either site-a node
|
||||
# 8 the same reference on site-b (deliberately mismatched dev KEK) -> deploy fails naming the
|
||||
# KekId, the CLI exits 1
|
||||
# 9 one DeploySiteSecrets audit row per deploy that carried, lacked or retired a secret (the
|
||||
# removal-only deploy of check 6 included, with its removed list); no plaintext / base64
|
||||
# ciphertext in any of the 8 nodes' logs
|
||||
#
|
||||
# Teardown (trap EXIT, also run up-front so a rerun starts clean): gate connections deleted,
|
||||
# site-a + site-b redeployed so the removal reaches them, the central gate secret deleted, the
|
||||
@@ -56,6 +61,10 @@ A=(--url "$URL" --format=json --username "${SCADABRIDGE_USERNAME:-multi-role}" -
|
||||
SECRETS_CLI_PROJECT="${SECRETS_CLI_PROJECT:-$HOME/Desktop/scadaproj/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Cli}"
|
||||
SECRETS_CLI_DIR="${SECRETS_CLI_DIR:-${TMPDIR:-/tmp}/sb-gate-secrets-cli}"
|
||||
STANDBY_WINDOW_S="${STANDBY_WINDOW_S:-30}"
|
||||
# Check 5: mirror pass (15 s) + status reporter poll (30 s) + health report (30 s), plus a restart.
|
||||
HEALTH_WINDOW_S="${HEALTH_WINDOW_S:-240}"
|
||||
# The site-b/site-c dev KEK (docker-compose.yml) — deliberately NOT site-a's, used by check 5.
|
||||
WRONG_KEK="esKrmqV2RBmMq/+6T/5zK2BcalRy5gzDXhoG9Qydg6Q="
|
||||
|
||||
# DEV-ONLY rig KEK shared by central + site-a (docker/docker-compose.yml, x-secrets-shared-kek-env).
|
||||
SITE_A_KEK="zZiBWuoaVMbJmGXToLk9Lakw0iJozXoL/7Gxac3GwJ4="
|
||||
@@ -333,6 +342,39 @@ v=None if r is None else r.get(sys.argv[2])
|
||||
print("" if v is None else (json.dumps(v) if isinstance(v,bool) else v))' "$1" "$2"
|
||||
}
|
||||
badge_of() { jget "$1" "$2"; }
|
||||
# mirror_failures <siteIdentifier> -> JSON {nodeName: [secretName...]} from central's latest health
|
||||
# report for the site (SiteHealthReport.SecretMirrorStatuses), or "null" when not reported
|
||||
mirror_failures() {
|
||||
"$CLI" health site "${A[@]}" --identifier "$1" 2>/dev/null | python3 -c '
|
||||
import json,sys
|
||||
try:
|
||||
r=(json.load(sys.stdin).get("latestReport") or {}).get("secretMirrorStatuses")
|
||||
except Exception:
|
||||
r=None
|
||||
print("null" if r is None else json.dumps({n["nodeName"]: sorted(f["secretName"] for f in n["failures"]) for n in r}, sort_keys=True))'
|
||||
}
|
||||
# The rig's compose project dir (where the bind-mounted appsettings live), from the running container.
|
||||
RIG_DIR=$(docker inspect "$STANDBY_A" --format '{{ index .Config.Labels "com.docker.compose.project.working_dir" }}' 2>/dev/null)
|
||||
# recreate_standby_a [wrong] -> recreates site-a-b from the rig's own compose file, optionally with
|
||||
# an override that swaps in a mismatched KEK; waits until it is a cluster member again.
|
||||
recreate_standby_a() {
|
||||
local files=(-f "$RIG_DIR/docker-compose.yml")
|
||||
if [ "${1:-}" = "wrong" ]; then
|
||||
printf 'services:\n site-a-b:\n environment:\n ZB_SECRETS_MASTER_KEY: "%s"\n' "$WRONG_KEK" >"$WORK/wrong-kek.yml"
|
||||
files+=(-f "$WORK/wrong-kek.yml"); touch "$WORK/standby-kek-swapped"
|
||||
fi
|
||||
docker compose "${files[@]}" --project-directory "$RIG_DIR" up -d --no-deps --force-recreate site-a-b >/dev/null 2>&1 || return 1
|
||||
[ "${1:-}" = "wrong" ] || rm -f "$WORK/standby-kek-swapped"
|
||||
local i
|
||||
for i in $(seq 1 60); do
|
||||
docker run --rm --network "container:$ACTIVE_A" python:3.13-alpine python3 -c '
|
||||
import urllib.request as u,sys
|
||||
try: sys.exit(0 if u.urlopen("http://scadabridge-site-a-b:8084/health/ready",timeout=3).status==200 else 1)
|
||||
except Exception: sys.exit(1)' >/dev/null 2>&1 && return 0
|
||||
sleep 3
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
ensure_secrets_cli() {
|
||||
[ -f "$SECRETS_CLI_DIR/ZB.MOM.WW.Secrets.Cli.dll" ] && return 0
|
||||
@@ -365,6 +407,9 @@ teardown() {
|
||||
echo ""
|
||||
echo "=== teardown ==="
|
||||
local id
|
||||
if [ -f "$WORK/standby-kek-swapped" ]; then
|
||||
recreate_standby_a && echo " restored site-a-b with site-a's KEK" || echo " WARN could not restore site-a-b's KEK"
|
||||
fi
|
||||
for c in "$CONN_A" "$CONN_B"; do
|
||||
id=$(conn_id "$c" 2>/dev/null)
|
||||
if [ -n "$id" ]; then
|
||||
@@ -493,7 +538,37 @@ B=$(gp badges "$URL")
|
||||
end_check 4
|
||||
|
||||
# ── 5 ────────────────────────────────────────────────────────────────────────────────────────
|
||||
begin_check 5 "drop the secret: reference -> deploy -> both nodes drop $SECRET; out-of-band $OOB survives"
|
||||
begin_check 5 "standby master-key mismatch reaches central's health report (site-a-b on a wrong KEK)"
|
||||
# wait_health <python predicate over F = {node: [failing secret names]}> -> sets HEALTH_S / LAST_F
|
||||
wait_health() {
|
||||
local start now; start=$(python3 -c 'import time; print(time.time())'); HEALTH_S=""
|
||||
while :; do
|
||||
LAST_F=$(mirror_failures site-a)
|
||||
if python3 -c "import json,sys; F=json.loads(sys.argv[1]); sys.exit(0 if F is not None and ($1) else 1)" "$LAST_F"; then
|
||||
now=$(python3 -c 'import time; print(time.time())'); HEALTH_S=$(python3 -c "print(round($now-$start))"); return 0
|
||||
fi
|
||||
now=$(python3 -c 'import time; print(time.time())')
|
||||
python3 -c "import sys; sys.exit(0 if $now-$start < $HEALTH_WINDOW_S else 1)" || return 1
|
||||
sleep 5
|
||||
done
|
||||
}
|
||||
if wait_health "F.get('node-a')==[] and F.get('node-b')==[]"; then ok "baseline: central reports both site-a nodes, no failures ($LAST_F)"
|
||||
else bad "baseline: site-a health report is not two clean node rows: $LAST_F"; fi
|
||||
if [ -z "$RIG_DIR" ]; then bad "cannot find the rig's compose directory from $STANDBY_A's labels"
|
||||
elif recreate_standby_a wrong; then
|
||||
ok "site-a-b recreated with a mismatched KEK and back in the cluster (standby; site-a-a stays active)"
|
||||
if wait_health "'$SECRET' in F.get('node-b',[]) and F.get('node-a')==[]"; then MISMATCH_S="$HEALTH_S"
|
||||
ok "central's site-a report shows node-b failing $SECRET, node-a clean (${HEALTH_S}s): $LAST_F"
|
||||
else bad "node-b's failure never reached central within ${HEALTH_WINDOW_S}s: $LAST_F"; fi
|
||||
if recreate_standby_a; then ok "site-a-b restored with site-a's KEK"; else bad "could not restore site-a-b"; fi
|
||||
if wait_health "F.get('node-a')==[] and F.get('node-b')==[]"; then CLEARED_S="$HEALTH_S"
|
||||
ok "node-b's failure cleared on central's report (${HEALTH_S}s)"
|
||||
else bad "node-b's failure did not clear within ${HEALTH_WINDOW_S}s: $LAST_F"; fi
|
||||
else bad "could not recreate site-a-b with a mismatched KEK"; fi
|
||||
end_check 5
|
||||
|
||||
# ── 6 ────────────────────────────────────────────────────────────────────────────────────────
|
||||
begin_check 6 "drop the secret: reference -> deploy -> both nodes drop $SECRET; out-of-band $OOB survives"
|
||||
SB_GATE_VALUE="$VALUE_OOB" site_a_secret_set "$OOB" "site-secrets gate out-of-band" >/dev/null 2>&1 || true
|
||||
J=$(gp site-secrets "$ACTIVE_A")
|
||||
if [ "$(python3 -c 'import json,sys; d=json.loads(sys.argv[1]); print(d.get(sys.argv[2],{}).get("deleted"))' "$J" "$OOB")" = "False" ]; then
|
||||
@@ -503,6 +578,7 @@ upsert_conn "$CONN_A" "$SITE_A_ID" "gate-literal-not-a-secret-ref" && ok "$CONN_
|
||||
T0=$(python3 -c 'import time; print(time.time())'); deploy "$SITE_A_ID"; T_DEPLOYED=$(python3 -c 'import time; print(time.time())')
|
||||
[ "$DEPLOY_RC" -eq 0 ] && [ "$(site_result site-a success)" = "true" ] && ok "deploy succeeded (secretsDeployed='$(site_result site-a secretsDeployed)')" \
|
||||
|| bad "deploy: rc=$DEPLOY_RC $(printf '%s' "$DEPLOY_OUT" | tr -d '\n' | cut -c1-240)"
|
||||
case "$(site_result site-a secretsRemoved)" in *"$SECRET"*) ok "reply names $SECRET in secretsRemoved";; *) bad "secretsRemoved='$(site_result site-a secretsRemoved)', expected $SECRET";; esac
|
||||
GONE="(d.get('$SECRET') is None or d['$SECRET']['deleted'])"
|
||||
if wait_node "$ACTIVE_A" "$GONE" "$T_DEPLOYED"; then ok "site-a-a no longer holds $SECRET (${WAIT_S}s)"; else bad "site-a-a still holds $SECRET: $LAST_NODE_JSON"; fi
|
||||
if wait_node "$STANDBY_A" "$GONE" "$T_DEPLOYED"; then STANDBY_5="$WAIT_S"; ok "site-a-b no longer holds $SECRET (${WAIT_S}s after the deploy returned)"
|
||||
@@ -512,10 +588,10 @@ J=$(gp site-secrets "$ACTIVE_A")
|
||||
&& ok "out-of-band $OOB still live on site-a-a" || bad "out-of-band $OOB lost: $J"
|
||||
L=$(gp site-localdb "$ACTIVE_A")
|
||||
[ "$(jget "$L" "deployed.$SECRET.removed")" = "true" ] && ok "deployed_secrets.$SECRET is_removed=1 (kept for the standby)" || bad "deployed_secrets: $L"
|
||||
end_check 5
|
||||
end_check 6
|
||||
|
||||
# ── 6 ────────────────────────────────────────────────────────────────────────────────────────
|
||||
begin_check 6 "reference non-existent $MISSING -> site-a deploy fails closed, stays Out of date, site untouched"
|
||||
# ── 7 ────────────────────────────────────────────────────────────────────────────────────────
|
||||
begin_check 7 "reference non-existent $MISSING -> site-a deploy fails closed, stays Out of date, site untouched"
|
||||
SNAP_BEFORE="$(site_hash site-a)|$(gp site-secrets "$ACTIVE_A")|$(gp site-localdb "$ACTIVE_A")|$(gp site-secrets "$STANDBY_A")|$(gp site-localdb "$STANDBY_A")"
|
||||
upsert_conn "$CONN_A" "$SITE_A_ID" "secret:$MISSING" && ok "$CONN_A now references secret:$MISSING" || bad "update $CONN_A"
|
||||
deploy "$SITE_A_ID"
|
||||
@@ -523,6 +599,7 @@ ERR=$(site_result site-a errorMessage)
|
||||
echo " deploy rc=$DEPLOY_RC error: $ERR"
|
||||
EXPECT="Secret '$MISSING' referenced by data connection(s) $CONN_A is not in the central secret store"
|
||||
[ "$(site_result site-a success)" = "false" ] && ok "site-a deploy reported failure" || bad "site-a deploy did not fail: $(printf '%s' "$DEPLOY_OUT" | tr -d '\n' | cut -c1-240)"
|
||||
[ "$DEPLOY_RC" -eq 1 ] && ok "CLI exited 1 (failureCount 1)" || bad "CLI exited $DEPLOY_RC, expected 1"
|
||||
case "$ERR" in *"$EXPECT"*) ok "named fail-closed message";; *) bad "message does not contain: $EXPECT";; esac
|
||||
B=$(gp badges "$URL"); echo " badges: $B"
|
||||
[ "$(badge_of "$B" site-a)" = "out of date" ] && ok "site-a stays Out of date" || bad "site-a is '$(badge_of "$B" site-a)'"
|
||||
@@ -530,10 +607,10 @@ sleep 3
|
||||
SNAP_AFTER="$(site_hash site-a)|$(gp site-secrets "$ACTIVE_A")|$(gp site-localdb "$ACTIVE_A")|$(gp site-secrets "$STANDBY_A")|$(gp site-localdb "$STANDBY_A")"
|
||||
[ "$SNAP_BEFORE" = "$SNAP_AFTER" ] && ok "nothing changed: stamped hash, both nodes' secret rows, deployed_secrets, ledgers and the $CONN_A definition identical" \
|
||||
|| { bad "site state changed"; echo " before: $SNAP_BEFORE"; echo " after: $SNAP_AFTER"; }
|
||||
end_check 6
|
||||
end_check 7
|
||||
|
||||
# ── 7 ────────────────────────────────────────────────────────────────────────────────────────
|
||||
begin_check 7 "the same reference on site-b (mismatched dev KEK) -> deploy fails naming the KekId"
|
||||
# ── 8 ────────────────────────────────────────────────────────────────────────────────────────
|
||||
begin_check 8 "the same reference on site-b (mismatched dev KEK) -> deploy fails naming the KekId"
|
||||
C=$(gp central-secret "$SECRET")
|
||||
KEK=$(jget "$C" kek)
|
||||
upsert_conn "$CONN_B" "$SITE_B_ID" "secret:$SECRET" && ok "created $CONN_B on site-b (ApiKey=secret:$SECRET)" || bad "create $CONN_B"
|
||||
@@ -541,13 +618,14 @@ deploy "$SITE_B_ID"
|
||||
ERR=$(site_result site-b errorMessage)
|
||||
echo " deploy rc=$DEPLOY_RC error: $ERR"
|
||||
[ "$(site_result site-b success)" = "false" ] && ok "site-b deploy reported failure" || bad "site-b deploy did not fail: $(printf '%s' "$DEPLOY_OUT" | tr -d '\n' | cut -c1-240)"
|
||||
[ "$DEPLOY_RC" -eq 1 ] && ok "CLI exited 1 (failureCount 1)" || bad "CLI exited $DEPLOY_RC, expected 1"
|
||||
case "$ERR" in *"$SECRET"*"$KEK"*) ok "message names $SECRET and the row's KekId $KEK";; *) bad "message does not name $SECRET + KekId $KEK";; esac
|
||||
B=$(gp badges "$URL"); echo " badges: $B"
|
||||
[ "$(badge_of "$B" site-b)" = "out of date" ] && ok "site-b not stamped (Out of date)" || bad "site-b is '$(badge_of "$B" site-b)'"
|
||||
end_check 7
|
||||
end_check 8
|
||||
|
||||
# ── 8 ────────────────────────────────────────────────────────────────────────────────────────
|
||||
begin_check 8 "DeploySiteSecrets audit rows; no plaintext / base64 ciphertext in any node log"
|
||||
# ── 9 ────────────────────────────────────────────────────────────────────────────────────────
|
||||
begin_check 9 "DeploySiteSecrets audit rows; no plaintext / base64 ciphertext in any node log"
|
||||
AUD=$("$CLI" audit-config query "${A[@]}" --action DeploySiteSecrets --entity-type SiteSecrets --from "$GATE_START" --page-size 50 2>/dev/null)
|
||||
ROWS=$(printf '%s' "$AUD" | python3 -c '
|
||||
import json,sys
|
||||
@@ -557,18 +635,20 @@ for e in sorted(d.get("entries",[]), key=lambda e: e["timestamp"]):
|
||||
s=json.loads(e.get("afterStateJson") or "{}")
|
||||
names=",".join("%s@r%s" % (x.get("name"), x.get("revision")) for x in (s.get("secrets") or []))
|
||||
miss=",".join(s.get("missing") or [])
|
||||
rows.append("%s|%s|%s|%s|%s" % (e["entityId"], s.get("success"), names or "-", miss or "-", s.get("deploymentId")))
|
||||
rem=",".join(s.get("removed") or [])
|
||||
rows.append("%s|%s|%s|%s|%s|%s" % (e["entityId"], s.get("success"), names or "-", miss or "-", rem or "-", s.get("deploymentId")))
|
||||
print("\n".join(rows))')
|
||||
echo "$ROWS" | sed 's/^/ audit: /'
|
||||
R2=$(jget "$C1" rev); R4=$(jget "$C2" rev)
|
||||
EXPECTED="site-a|True|$SECRET@r$R2|-
|
||||
site-a|True|$SECRET@r$R4|-
|
||||
site-a|False|-|$MISSING
|
||||
site-b|False|$SECRET@r$R4|-"
|
||||
GOT=$(printf '%s\n' "$ROWS" | cut -d'|' -f1-4)
|
||||
[ "$GOT" = "$EXPECTED" ] && ok "exactly 4 rows, one per secret-bearing deploy (2, 4, 6-fail-closed, 7-kek-mismatch); the no-secret deploy of check 5 wrote none" \
|
||||
EXPECTED="site-a|True|$SECRET@r$R2|-|-
|
||||
site-a|True|$SECRET@r$R4|-|-
|
||||
site-a|True|-|-|$SECRET
|
||||
site-a|False|-|$MISSING|-
|
||||
site-b|False|$SECRET@r$R4|-|-"
|
||||
GOT=$(printf '%s\n' "$ROWS" | cut -d'|' -f1-5)
|
||||
[ "$GOT" = "$EXPECTED" ] && ok "exactly 5 rows: deploys 2 and 4, the removal-only deploy of check 6 (removed=$SECRET), 7 fail-closed, 8 kek-mismatch" \
|
||||
|| { bad "audit rows differ from expected"; echo " expected:"; echo "$EXPECTED" | sed 's/^/ /'; }
|
||||
[ "$(printf '%s\n' "$ROWS" | cut -d'|' -f5 | sort -u | grep -c .)" = "4" ] && ok "4 distinct deploymentIds" || bad "deploymentIds not distinct"
|
||||
[ "$(printf '%s\n' "$ROWS" | cut -d'|' -f6 | sort -u | grep -c .)" = "5" ] && ok "5 distinct deploymentIds" || bad "deploymentIds not distinct"
|
||||
# Material to hunt for: both plaintext values and both ciphertexts (base64, as they appear in any
|
||||
# JSON serialization). Written to a pattern file so they never reach argv or the output.
|
||||
PAT="$WORK/patterns"; : >"$PAT"; chmod 600 "$PAT"
|
||||
@@ -584,10 +664,11 @@ done
|
||||
AUDH=$(printf '%s' "$AUD" | grep -c -F -f "$PAT" || true)
|
||||
[ "$HITS" = "0" ] && ok "0 hits across all 8 nodes' logs since $LOG_SINCE" || bad "$HITS log line(s) carry secret material"
|
||||
[ "$AUDH" = "0" ] && ok "0 hits in the DeploySiteSecrets audit payloads" || bad "audit payload carries secret material"
|
||||
end_check 8
|
||||
end_check 9
|
||||
|
||||
echo ""
|
||||
echo "=== Summary ==="
|
||||
echo "standby convergence (site-a-b, seconds after the deploy call returned): check3=${STANDBY_3:-?} check4=${STANDBY_4:-?} check5=${STANDBY_5:-?}"
|
||||
echo "PASS $CHECKS_PASSED/8"
|
||||
echo "standby convergence (site-a-b, seconds after the deploy call returned): check3=${STANDBY_3:-?} check4=${STANDBY_4:-?} check6=${STANDBY_5:-?}"
|
||||
echo "standby mismatch on central's health report (check 5): shown after ${MISMATCH_S:-?}s, cleared after ${CLEARED_S:-?}s"
|
||||
echo "PASS $CHECKS_PASSED/9"
|
||||
[ "$CHECKS_FAILED" -eq 0 ] || exit 1
|
||||
|
||||
@@ -120,7 +120,7 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round
|
||||
| 124 | **No template-graph-watermark short-circuit on the pending-updates fleet flatten** *(Low)*. `PendingUpdatesEvaluator.EvaluateAsync` re-flattens every deployed instance through `DeploymentService.GetInstanceStalenessAsync` on every cache-TTL expiry (`PendingUpdatesCacheTtl`, default 60 s) while any Deployer/Admin circuit is open, even when nothing in the template/instance/artifact graph has changed since the last evaluation. | `docs/plans/2026-09-14-pending-updates-design.md` §3.2 ("Not in scope") | The single-flight TTL cache already bounds the cost to one fleet flatten per TTL fleet-wide (not per viewer); a watermark short-circuit is a further optimization, not a correctness need, and needs a cheap "has anything changed" signal that does not exist yet. | The fleet flatten's cost (instance count x template depth) becomes measurable against `PendingUpdatesCacheTtl` on a large deployment. |
|
||||
| 125 | **Pending-updates counts have no Health-dashboard tile or CLI surface** *(Low)*. The aggregate (`PendingUpdatesSnapshot.StaleInstanceCount` / `StaleArtifactSiteCount`) is consumed only by the Central UI Sites page and rail indicator; there is no Health dashboard KPI tile and no `deploy`/`site` CLI command surfacing it — `site list` shows only the raw `artifactRevisionHash`/`artifactsDeployedAt` fields on each site, not whether they are current. | `docs/plans/2026-09-14-pending-updates-design.md` §3.2 ("Not in scope") | Out of scope for the UI-focused ask; the CLI/Health surfaces are a separate consumer of the same `IPendingUpdatesCache`. | An operator wants the pending-updates counts without opening the Central UI (headless monitoring, CLI scripting). |
|
||||
| 126 | ~~**TODO — roll the two outstanding migrations to the QA pair vq03/vq04**~~ **DONE 2026-09-28** — QA vq03/vq04 upgraded `2b9a7c3c` → `b3021e7c`, 55 → 57, artifacts deployed (record `docs/deployment/deployment-records/2026-09-28-wonder-app-vq03-vq04-b3021e7c.md`). *(deployment, closed)*. **DEV vd03 DONE 2026-09-28** (`fe89363b`, 55 → 57, record `docs/deployment/deployment-records/2026-09-28-wonder-app-vd03-fe89363b.md` — the `sb-mig-55to57.sql` there is the same script QA needs). QA still runs `2b9a7c3c` with the DB at migration 55 (`20260909203046_EnforceGlobalInstanceNamesAndProtectProtocolConfiguration`); `main` (`ac833f5a`, pushed 2026-09-14) carries two more: **56** `20260911135501_AddAlarmMesReportable` (computed-alarm `IsMesReportable` flag) and **57** `20260914173147_AddSiteArtifactRevision` (`Sites.ArtifactRevisionHash`/`ArtifactsDeployedAt`, both nullable). Both are additive `AddColumn`s — no data upgrade, no coordinated step like the 2026-09-10 protocol-config encryption. Recipe (same as the 55 rollout): generate ONE from→to delta script `dotnet ef migrations script EnforceGlobalInstanceNamesAndProtectProtocolConfiguration AddSiteArtifactRevision --project src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase --startup-project src/ZB.MOM.WW.ScadaBridge.Host -o sb-mig-55to57.sql` (NOT idempotent, identical for vd03 and the QA pair), take the pre-upgrade DB backup, stop the pair TOGETHER, apply via `sqlcmd -C -b -I` on the SQL host, publish/swap the `ac833f5a` binaries, start the pair, confirm the Central boot log shows 0 pending migrations and no "Applying migration" (the script already applied them). After upgrade every site reads *Artifacts never deployed* until one *Deploy Artifacts to All Sites* — run it as the last step. Write the deployment record `docs/deployment/deployment-records/<date>-wonder-app-{vd03,vq03-vq04}-ac833f5a.md`. | Pending-updates finish (2026-09-14) | ~~Blocked by the 2026-09-11 remote-access hold~~ — **unblocked 2026-09-28**: `winrm` is re-enabled for deployment activities (CLAUDE.md). vd03 done; QA done 2026-09-28. | Closed 2026-09-28. |
|
||||
| 127 | **TODO — roll out site secrets via artifact deploy to vd03 + QA (vq03/vq04)**. The gRPC secrets hub is retired in ScadaBridge as of `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md` (implemented on branch `feat/site-secrets-artifact-deploy`, rig-only so far); vd03 and QA still run the pre-feature build with `Secrets:Replication:Enabled=false`, so upgrading them will **not** fail the boot (the retirement's `StartupValidator` rejection only fires on `Enabled=true`). **Upgrade impact:** once upgraded, any site whose data connections carry `secret:` references fails its **whole** artifact deploy (scripts, external systems, DB connections too) and reads *Out of date* until every referenced secret exists in the central store — so before upgrading vd03/QA, check whether their connection configs use `secret:` references and whether their site pairs run LocalDb replication, and do the prerequisites in the same window. Prerequisites, in order: (1) provision the `zbsecrets` schema/store in the environment's configuration SQL Server (the same server `ConfigurationDb` already points at); (2) set `Secrets:Store:Provider=SqlServer` on **both** central nodes of the environment; (3) verify the KEK (`ZB_SECRETS_MASTER_KEY` / master-key file) is identical across **every** node — both centrals and every site node — since a mismatch is caught only at deploy time by `DeployedSecretsMirror`'s per-secret verify, not at boot; (4) confirm every site pair that uses secrets runs LocalDb replication — on an unreplicated pair only the node active at deploy time gets the secrets, so a failover leaves the new active node without them until the next *Deploy Artifacts*; (5) recreate, at the new central shared store, any secret an operator previously hand-seeded directly into a site's local `secrets.db` (those never round-tripped through central and would otherwise be silently absent from the referenced-secrets scan). Then run *Deploy Artifacts* against every affected site as the rollout's last step. | `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md` §4 *Rollout*; `docs/deployment/topology-guide.md` → *Secrets: shared store, per-site deploy, and the KEK* | Deployment record, not a code change — vd03/QA config is user-held and their rollout schedule is independent of the branch merging to main. | User asks for the vd03/QA secrets window. |
|
||||
| 127 | **TODO — roll out site secrets via artifact deploy to vd03 + QA (vq03/vq04)**. The gRPC secrets hub is retired in ScadaBridge as of `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md` (merged to `main` @ `23a139a4` 2026-09-28, plus the same-day follow-ups: per-site secret-store outage handling, removal audit rows, per-node mirror status on the health report, artifact-deploy CLI exit codes; rig-only so far); vd03 and QA still run the pre-feature build with `Secrets:Replication:Enabled=false`, so upgrading them will **not** fail the boot (the retirement's `StartupValidator` rejection only fires on `Enabled=true`). **Upgrade impact:** once upgraded, any site whose data connections carry `secret:` references fails its **whole** artifact deploy (scripts, external systems, DB connections too) and reads *Out of date* until every referenced secret exists in the central store — so before upgrading vd03/QA, check whether their connection configs use `secret:` references and whether their site pairs run LocalDb replication, and do the prerequisites in the same window. Prerequisites, in order: (1) provision the `zbsecrets` schema/store in the environment's configuration SQL Server (the same server `ConfigurationDb` already points at); (2) set `Secrets:Store:Provider=SqlServer` on **both** central nodes of the environment; (3) verify the KEK (`ZB_SECRETS_MASTER_KEY` / master-key file) is identical across **every** node — both centrals and every site node — since a mismatch is caught only at deploy time by `DeployedSecretsMirror`'s per-secret verify, not at boot (after the first deploy a mismatch on either node — the standby included — shows as a *Secret mirror failing* badge on that site's Health card); (4) confirm every site pair that uses secrets runs LocalDb replication — on an unreplicated pair only the node active at deploy time gets the secrets, so a failover leaves the new active node without them until the next *Deploy Artifacts*; the upgrade also adds two replicated site tables (`deployed_secrets`, `secret_mirror_status`), so stop and start each site pair together as usual; (5) recreate, at the new central shared store, any secret an operator previously hand-seeded directly into a site's local `secrets.db` (those never round-tripped through central and would otherwise be silently absent from the referenced-secrets scan). Then run *Deploy Artifacts* against every affected site as the rollout's last step. | `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md` §4 *Rollout*; `docs/deployment/topology-guide.md` → *Secrets: shared store, per-site deploy, and the KEK* | Deployment record, not a code change — vd03/QA config is user-held and their rollout schedule is independent of the branch merging to main. | User asks for the vd03/QA secrets window. |
|
||||
|
||||
## Resolved (verified against the code 2026-07-10)
|
||||
Rows removed from the Deferred table above once confirmed shipped. Kept here for traceability.
|
||||
|
||||
@@ -215,11 +215,10 @@ accurate as the spec:
|
||||
Grpc mode) — without that fix a central's boot-time `${secret:}` tokens and its running
|
||||
`ISecretResolver` could resolve a token differently.
|
||||
- **`ZB.MOM.WW.Secrets` 0.6.3 (hub sweep logging, §3.7's third bullet — `EventId`s 4/5/6,
|
||||
`ISecretsHubSyncStatus`) was implemented and tested on a `scadaproj` branch but is NOT
|
||||
published to the package feed, and ScadaBridge does not consume it.** Publishing is an outward
|
||||
side effect that needs explicit user consent, and it would be a dead pin here regardless —
|
||||
hub retirement (above) removes ScadaBridge's only dependency on the hub client entirely. It
|
||||
remains available for the library's other hub consumers once published.
|
||||
`ISecretsHubSyncStatus`) is published to the Gitea feed (2026-09-28, after merge), but
|
||||
ScadaBridge does not consume it** — hub retirement (above) removes ScadaBridge's only dependency
|
||||
on the hub client, so it stays on 0.6.2. It is there for the library's other hub consumers
|
||||
(OtOpcUa).
|
||||
- **Empty, not null, when a site references no secrets (Task 13 live-gate fix).** The command
|
||||
builder first shipped `Secrets = null` for a site with no references ("null-when-empty", borrowed
|
||||
from the hash convention). But the site treats `null` as "central shipped no secrets" and touches
|
||||
@@ -237,11 +236,41 @@ accurate as the spec:
|
||||
next (re)connect (no forced reconnect).
|
||||
- *One `secret:` parser* (`SecretReferenceScanner.TryParseSecretReference`) shared by the scanner
|
||||
and `MxGatewayDataConnection`, so a padded/case-variant reference resolves the shipped name.
|
||||
- *Audit shape (§3.7 deviation).* The `DeploySiteSecrets` row carries `secrets` and `missing` —
|
||||
there is **no `removed` field** — and a removal-only deploy (empty list shipped) writes **no
|
||||
central row**; removals are logged on the site by `DeployedSecretsMirror`.
|
||||
- *Audit shape (§3.7).* The `DeploySiteSecrets` row carries `secrets`, `missing` and `removed`.
|
||||
(Shipped first without `removed`, so a removal-only deploy wrote no central row; closed by the
|
||||
follow-ups below.)
|
||||
- *Deployment prerequisite:* a site pair using deployed secrets must run LocalDb replication
|
||||
(topology guide), since on an unreplicated pair only the node active at deploy time gets them.
|
||||
- **Follow-ups closed after merge (2026-09-28).**
|
||||
- *Secret-store outage is per site (review M5).* `SiteSecretsCollector` returns a store read
|
||||
failure as `SiteSecretsCollection.StoreError` instead of throwing. A deploy fails only the
|
||||
sites that reference a secret (nothing is sent to them; they get a `DeploySiteSecrets` row
|
||||
with the error), and `ComputeCurrentArtifactHashesAsync` leaves those sites out of its map,
|
||||
which the pending-updates evaluator reads as *Out of date*. Sites with no secret references
|
||||
never touch the store and are unaffected. Before this, one `GetAsync` exception aborted the
|
||||
whole sweep and the fleet-wide pending-updates check.
|
||||
- *Removal-only deploys are audited.* The site's reply gained `SecretsRemoved` (proto
|
||||
`ArtifactDeploymentResponseDto.secrets_removed = 7`, names only; absent/empty reads as null):
|
||||
the `deployed_secrets` rows that deploy soft-removed. Central carries it on
|
||||
`SiteArtifactResult.SecretsRemoved` and writes the `DeploySiteSecrets` row whenever it is
|
||||
non-empty, with a `removed` list. An old site never sends it, so its removals stay unaudited
|
||||
centrally, as before.
|
||||
- *CLI exit code.* `site deploy-artifacts` and `deploy artifacts` exit `1` when the summary's
|
||||
`failureCount` is non-zero (same rule `deploy site` already had).
|
||||
- *Hub-era site-a secrets (review M8).* The retired hub had copied six `hub-gate-*`/`shared-gate-*`
|
||||
test secrets (2026-08-07) into both rig site-a nodes' local stores. The mirror deliberately
|
||||
never deletes a name it did not ledger, so they were removed once by hand on the rig (both
|
||||
nodes stopped together; see the live-gate doc). QA and vd03 never ran the hub
|
||||
(`Secrets:Replication:Enabled=false`), so nothing like this exists there.
|
||||
- **Standby mirror failures reach central (follow-up, review finding M1).** A `DeployedSecretsMirror`
|
||||
failure on the standby (e.g. a KEK mismatch) was only logged on that node — site health reports
|
||||
are sent by the active node only. Each node now records its current failure set (names +
|
||||
messages, never values) in a new, **12th LocalDb-replicated table `secret_mirror_status`** (one
|
||||
row per node keyed by `node_name`), written only when the set changes plus once per start-up; a
|
||||
per-node `SecretMirrorStatusReporter` puts every node's row on the active node's report as
|
||||
`SiteHealthReport.SecretMirrorStatuses` (proto `SiteHealthReportDto` field 33, presence-wrapped),
|
||||
and the Central UI health card shows a *Secret mirror failing* badge + per-node list. Requires
|
||||
LocalDb replication for the standby's row to be visible, the same prerequisite as the secrets.
|
||||
- **Rig posture.** `docker/docker-compose.yml`'s central nodes now carry
|
||||
`Secrets__Store__Provider=SqlServer` plus a shared KEK; site-a shares that KEK (it is the
|
||||
secrets-via-artifact-deploy site); site-b/site-c deliberately carry a **mismatched** KEK so the
|
||||
|
||||
@@ -151,3 +151,46 @@ rows, 0 plaintext/ciphertext hits across all 8 node logs and the audit payloads.
|
||||
|
||||
Tests before the rig run: `dotnet build ZB.MOM.WW.ScadaBridge.slnx` 0 errors; Commons.Tests 875/875,
|
||||
SiteRuntime.Tests 892/892, DeploymentManager.Tests 235/235, DataConnectionLayer.Tests 383/383, Host.Tests 593/593.
|
||||
|
||||
## 7. Follow-ups run (2026-09-28)
|
||||
|
||||
Code under test: branch `fix/site-secrets-followups` — per-site secret-store outage handling and
|
||||
removal-only audit rows (`29049c4e`), artifact-deploy CLI exit codes (`ef03a0dc`), per-node mirror status on
|
||||
the site health report (`0289d511`, merged in `4e640918`). Image rebuilt from the worktree with
|
||||
`bash docker/build.sh`, all 8 nodes recreated from the primary checkout's compose file.
|
||||
|
||||
**Hub-era secrets removed from site-a (review M8), before the run.** The retired gRPC hub had left nine rows
|
||||
dated 2026-08-07 in both site-a nodes' local stores (`/app/data/scadabridge-secrets.db`, table `secret`):
|
||||
six live (`hub-gate-smoke-2`, `hub-gate-smoke-3`, `shared-gate-051`, `shared-gate-2`, `shared-gate-2b`,
|
||||
`shared-gate-final`) and three tombstones (`expander-gate-pepper`, `hub-gate-smoke-1`, `shared-gate-1`).
|
||||
Nothing in the rig config referenced them. With the site-a pair stopped together, each node's file was edited
|
||||
from a throwaway `python:3.13-alpine` container mounting that node's data volume (`DELETE … WHERE name IN (…)
|
||||
AND updated_utc < '2026-08-08'`, then `wal_checkpoint(TRUNCATE)`); 9 rows deleted per node, only the `sb-gate-*`
|
||||
gate tombstones left. Both nodes came back ready, site-a-a active. site-b/site-c never ran the hub and had no
|
||||
such rows; QA and vd03 never enabled replication.
|
||||
|
||||
**The gate grew to nine checks:**
|
||||
- new check 5 (M1): site-a-b is recreated with the site-b/site-c dev KEK (a compose override, removed again in
|
||||
the same check and in teardown), and central's `health site --identifier site-a` must show node-b failing
|
||||
`sb-gate-mxgw` while node-a is clean, then both clean once site-a-b has site-a's KEK again;
|
||||
- check 6 (was 5) also asserts the reply's `secretsRemoved` names the dropped secret;
|
||||
- checks 7 and 8 (were 6 and 7) also assert the CLI exits 1;
|
||||
- check 9 (was 8) now expects five `DeploySiteSecrets` rows — the removal-only deploy of check 6 writes one,
|
||||
with `removed: [sb-gate-mxgw]`.
|
||||
|
||||
`bash docker/verify-site-secrets.sh` → **PASS 9/9**. Before the check, central reported both site-a node
|
||||
rows with no failures; site-b and site-c (unreplicated) report their active node only. With the wrong KEK,
|
||||
node-b's `sb-gate-mxgw` failure appeared on central's site-a report 57 s after site-a-b was back in the cluster
|
||||
(mirror pass 15 s + status reporter 30 s + health report 30 s), and cleared 57 s after the KEK was restored.
|
||||
Standby convergence: check 3 = 4.3 s, check 4 = 4.2 s, check 6 = 15.1 s. Audit: the 5 expected rows, 5 distinct
|
||||
deployment ids; 0 plaintext/ciphertext hits in all 8 node logs and the audit payloads. Teardown left all three
|
||||
sites *Current* and site-a-b on site-a's KEK.
|
||||
|
||||
Not exercised live: the central secret-store outage (review M5) — taking the shared SQL store down on the rig
|
||||
also takes down central's configuration database. Covered by unit tests (`SiteSecretsCollectorTests`,
|
||||
`ArtifactDeploymentServiceTests.StoreOutage_*`, `ComputeCurrentHashes_StoreOutage_*`,
|
||||
`RetryForSite_StoreOutage_*`).
|
||||
|
||||
Tests: `dotnet build ZB.MOM.WW.ScadaBridge.slnx` 0 errors / 0 warnings; every non-Playwright test project green
|
||||
(CLI.Tests 553 after updating `DeploymentDeadlineParityTests`' stub server to reply with a real zero-failure
|
||||
summary — a bare `{}` now reads as an invalid artifact-deploy summary and exits 1). Playwright not run.
|
||||
|
||||
@@ -291,6 +291,7 @@ Per-leaf alarm rendering (leaf nodes are individual conditions for native alarms
|
||||
- Headline **Notification Outbox KPI tiles** — queue depth, stuck count, and parked count. These are central-computed by the Notification Outbox from the central `Notifications` table (not part of any site health report). The full outbox view is on the dedicated Notification Outbox page.
|
||||
- Headline **Site Call Audit KPI tiles** — buffered count, parked count, and failed-last-interval. These are central-computed by the Site Call Audit component from the central `SiteCalls` table (not part of any site health report). The full cached-call view is on the dedicated Site Calls page.
|
||||
- Headline **Audit KPI tiles** — three tiles in a new "Audit" KPI group: **Audit volume**, **Audit error rate**, and **Audit backlog**. These are sourced from the Audit Log component (#23) and Health Monitoring per the metric definitions in Component-HealthMonitoring.md; the dashboard simply surfaces them. The full audit query view is on the dedicated Audit Log page.
|
||||
- **Secret mirror failing** (review finding M1) — a warning badge on an online site's card when any node's `SiteHealthReport.SecretMirrorStatuses` entry lists failures, plus a compact warning panel at the top of the card body listing *node · secret: message* for each failure. The standby's failures appear here too (they ride the active node's report), so a master-key mismatch on the standby is visible before a failover exposes it. Null (no data) and all-healthy nodes render nothing; offline sites are skipped like the other warning badges.
|
||||
- **Trigger failover** (`CentralFailoverControl`, Administrator only) — on the central card and on each site card. Disabled with an explanatory tooltip when there is no standby to take over. It always confirms first, and the confirmation's red button reads **Fail over** (see the dialog-label contract under Technology); the central variant's copy states that this page is served by the node being restarted and will disconnect and reconnect, while the site variant deliberately does not, since a site is a different cluster.
|
||||
- **Shared KPI cache (arch-review WP2.4).** All four KPI families above — Notification Outbox, Site Call Audit, Audit, and their per-site/per-node breakdowns — are read through a process-level `IKpiSnapshotCache`, not queried per page load. Each accessor is independently memoized (8s TTL) with single-flight production, so N Blazor circuits polling the same KPI inside one window (e.g. ten operators with the Health dashboard open) cost the node ONE aggregate SQL round trip per KPI per window, not N. A failed query is never memoized — the next caller re-attempts it, and the calling page's existing per-tile "unavailable" degradation is unchanged. Every KPI-tile page (Health, Notification Outbox, Site Calls, Audit Log) shares the same cache instance.
|
||||
|
||||
|
||||
@@ -385,16 +385,25 @@ out below.
|
||||
check 5). An old site ignores proto field 9, so the empty list costs it
|
||||
nothing, and the hash is unaffected (empty == null above).
|
||||
- **Audit.** One best-effort `DeploySiteSecrets` / `SiteSecrets` audit row is
|
||||
written per site deploy that carried secrets or a missing-secret failure
|
||||
(never for a site with no secret references at all) — after-state is `{
|
||||
deploymentId, secrets: [{name, revision}], missing: [name], success }`, which
|
||||
structurally excludes ciphertext. Written on both `DeployCoreAsync` and
|
||||
`RetryForSiteAsync`, including the fail-closed path (audited even though
|
||||
nothing was sent). The row carries `secrets` and `missing` only — there is
|
||||
**no `removed` field** (design §3.7 named one), and a deploy that only
|
||||
removes secrets (the site's last reference dropped, an empty list shipped)
|
||||
writes **no central row**; removals are logged on the site by
|
||||
`DeployedSecretsMirror`.
|
||||
written per site deploy that carried secrets, failed on a missing secret or a
|
||||
secret-store read, or retired previously-deployed secrets (never for a site
|
||||
with no secret involvement at all) — after-state is `{ deploymentId,
|
||||
secrets: [{name, revision}], missing: [name], removed: [name], success,
|
||||
error }`, which structurally excludes ciphertext. Written on both
|
||||
`DeployCoreAsync` and `RetryForSiteAsync`, including the fail-closed paths
|
||||
(audited even though nothing was sent). `removed` is the site's reply field
|
||||
`ArtifactDeploymentResponse.SecretsRemoved` — the `deployed_secrets` rows that
|
||||
deploy soft-removed — so a deploy that only drops the site's last secret
|
||||
reference is still audited centrally. A site that predates the field never
|
||||
reports it.
|
||||
- **Central secret-store outage.** `SiteSecretsCollector` returns a store read
|
||||
failure as `StoreError` rather than throwing, so it is contained per site: a
|
||||
site that references a secret fails (`Central secret store could not be read
|
||||
while resolving '<name>' (...)`, nothing sent, not stamped), every other site
|
||||
deploys, and `ComputeCurrentArtifactHashesAsync` omits only the affected sites
|
||||
from its map (they read *Out of date* on the Sites page) instead of failing the
|
||||
fleet-wide pending-updates check. A site with no secret references never
|
||||
touches the store.
|
||||
- **Secret rotation and live connections.** The site writes and mirrors the
|
||||
shipped secrets before it pushes data connections to the DCL, so a
|
||||
connection created by the same deploy resolves a new secret on first
|
||||
|
||||
@@ -39,6 +39,7 @@ Site clusters (metric collection and reporting). Central cluster (aggregation an
|
||||
| `AuditRedactionFailure` | Audit Log (central) | Count of payload redactor errors (over-redacted payloads, safety-net hit) since the last interval. |
|
||||
| `LocalDbReplicationConnected` | Consolidated site LocalDb | Whether a peer sync session is currently running on this node. **Nullable — null means "no reading", not "disconnected".** Reported only by a node with replication configured; an unconfigured node reports **null**, never `false` (register row 80 — see *Unconfigured replication reports null* below). |
|
||||
| `StartupReconcilerActive` | Host (site) | Whether this node spawned its `SiteReconciliationActor` at boot. **Tri-state:** null = not reported (the central self-report, or a pre-row-73 site build), `true` = running, `false` = the node booted with no `IDeploymentConfigFetcher` and therefore has **no startup self-heal**. Latched once at boot by the Host — not polled. Drives the *No self-heal* dashboard badge (register row 73). |
|
||||
| `SecretMirrorStatuses` | Site Runtime (`DeployedSecretsMirror` via the replicated `secret_mirror_status` table) | Every site node's deployed-secrets mirror status — node name, when its failure set last changed, and its current failures (`{SecretName, Message}`; names and messages only, never values). Review finding M1: the **standby's** failures ride the **active** node's report, because each node writes its own row to a LocalDb-replicated table and the active node's `SecretMirrorStatusReporter` reads them all. **Nullable — null = no data** (central self-report, a pre-M1 site, reporter not yet run); an empty list = wired, no row yet; a node with an empty `Failures` list is healthy. Point-in-time (not drained on collect). Wire: `SiteHealthReportDto.secret_mirror_statuses` (field **33**, a presence wrapper so null ≠ empty). Drives the *Secret mirror failing* dashboard badge + per-node failure list. On an unreplicated pair only the active node's own row appears. |
|
||||
| `LocalDbOplogBacklog` | Consolidated site LocalDb | Unacked oplog depth (Prometheus `localdb_oplog_depth`). **Nullable — null means "no reading", NOT "connected with an empty backlog"** (a failed poll rendered as 0 would report a pair that cannot read its own oplog as perfectly healthy). **Expected transient spikes:** the daily `site_events` retention purge and the storage-cap trim are CDC-captured like any other write, so the backlog jumps by the batch size at purge time and drains as the peer acks it — see `Component-SiteEventLogging.md` → Storage and `docs/deployment/topology-guide.md` → *Reading the replication backlog* for the healthy-versus-faulty signature. |
|
||||
|
||||
## Reporting Protocol
|
||||
@@ -232,6 +233,7 @@ endpoint (`MetricsPort`, default 8084) rather than sent to central.
|
||||
- **Communication Layer**: Transports health reports from sites to central.
|
||||
- **Data Connection Layer (site)**: Provides connection health metrics.
|
||||
- **Site Runtime (site)**: Provides script error rate and alarm evaluation error rate metrics.
|
||||
- **Site Runtime (site) — deployed secrets**: Provides `SecretMirrorStatuses` via its own `SecretMirrorStatusReporter` hosted service (SiteRuntime already references HealthMonitoring and owns the `secret_mirror_status` storage, following the `SiteStreamAlarmDropReporter` precedent).
|
||||
- **Store-and-Forward Engine (site)**: Provides buffer depth metrics, including the notification backlog awaiting forward to central.
|
||||
- **Cluster Infrastructure (site)**: Provides node role status.
|
||||
- **Host (site)**: Supplies the two boot-time facts Health Monitoring must not derive for itself — the LocalDb replication-configured gate behind `LocalDbReplicationConnected`/`LocalDbOplogBacklog` (from `SiteLocalDbSetup.ReplicationIsConfigured`), and `StartupReconcilerActive`, latched where the Host decides whether to spawn `SiteReconciliationActor`. Both follow the `SiteEventLogReplicationCheck` precedent: the Host owns the configuration reading, the component owns the reporting.
|
||||
|
||||
@@ -166,6 +166,7 @@ flowchart TD
|
||||
- `HandleDeployArtifacts` additionally accepts the command's `Secrets` list (`SecretArtifact`, ciphertext only) and writes the full shipped set to the `deployed_secrets` table via `SiteStorageService.UpsertDeployedSecretsAsync` — upsert every shipped name, soft-remove (`is_removed = 1`) any stored name absent from the set. A `null` `Secrets` field (pre-feature central) touches nothing and answers with no `SecretsApplied`. A current central always ships a list — **empty** when the site references no secrets — so dropping a site's last `secret:` reference removes the secret from both nodes. Storage detail: *Deployed Secrets (Site SQLite / LocalDb)* below.
|
||||
- After writing the table, the active node runs `DeployedSecretsMirror.ReconcileAsync` **synchronously**, before replying — so the deploy response's `SecretsApplied` count and any apply/verify error reflect this deploy's own outcome, not a value left over from the mirror's background schedule. `SecretsApplied` counts only the names *this deploy shipped* that applied and verified cleanly; a mirror failure for an unrelated, already-broken name (a different deploy's leftover, or a stale removal) is logged at Warning and excluded — it neither fails this deploy nor reduces its `SecretsApplied`.
|
||||
- No `DeployedSecretsMirror` registered (secrets deployment not composed on this node) with a non-empty `Secrets` list fails the deploy with `Secret deployment is not enabled on this site node`; with an empty `Secrets` list it still succeeds (`SecretsApplied = 0`).
|
||||
- The reply also carries `SecretsRemoved`: the names `UpsertDeployedSecretsAsync` soft-removed on this deploy (site-wide, whichever node's mirror later deletes them), or `null` when none. Central uses it for the `removed` list of its `DeploySiteSecrets` audit row, so a deploy that only retires secrets is audited centrally.
|
||||
- **Ordering against the DCL.** The shipped secrets are written and mirrored **before** the deploy's data-connection definitions are pushed to the DCL (the definitions are still stored in SQLite at the same point, and the push still goes out if the secrets step fails), so a connection created or reconfigured by the same deploy resolves a newly shipped secret on its first connect. A rotation with **unchanged** connection config does not recreate the connection (the DCL push is hash-guarded) — the new value takes effect on that connection's next (re)connect. No forced reconnect is performed.
|
||||
- See *Deployed Secrets (Site SQLite / LocalDb)* below for the mirror itself.
|
||||
|
||||
@@ -433,9 +434,10 @@ The Instance Actor owns native-alarm setup alongside its computed Script and Ala
|
||||
|
||||
## Deployed Secrets (Site SQLite / LocalDb)
|
||||
|
||||
Site secrets ride the artifact deploy (design `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`; see that doc's *Implementation notes (as built)* for deviations from the original design). Two tables, two different replication postures:
|
||||
Site secrets ride the artifact deploy (design `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`; see that doc's *Implementation notes (as built)* for deviations from the original design). Three tables, two different replication postures:
|
||||
|
||||
- **`deployed_secrets`** — the **11th LocalDb-replicated table** (`SiteLocalDbSetup.ReplicatedTables`, up from ten). Columns: `name` (PK), `row_json` (the shipped `SecretArtifact` — ciphertext, nonce, tag, wrapped DEK, `KekId`, revision; never plaintext), `revision`, `updated_utc`, `is_removed` (soft-remove flag, default 0), `deployment_id`, `applied_at`. No FKs (the standing FK guard — `SiteLocalDbReplicatedTableFkGuardTests` — applies to it like every other replicated table). `UpsertDeployedSecretsAsync` is full-set upsert-then-soft-remove, never a hard `DELETE`, so a removal is itself a replicated row change the standby node can react to.
|
||||
- **`secret_mirror_status`** — the **12th LocalDb-replicated table** (review finding M1). One row per site node: `node_name` (PK — `INodeIdentityProvider.NodeName`, i.e. `node-a`/`node-b`, falling back to the machine name when unconfigured), `failures_json` (a JSON array of `{SecretName, Message}` — the node's *current* mirror failures; names and messages only, never values or ciphertext), `checked_at`. No FKs. It is per-node *data* but replicated on purpose: each node writes only its own row, and replication is what lets the **active** node read the **standby's** row onto the site health report (only the active node sends one — see *Mirror status on the health report* below).
|
||||
- **`mirrored_secrets`** — a **node-local, unreplicated** ledger (`name` PK, `revision`, `mirrored_at`) recording which names *this specific node* has applied into its own `ISecretStore`. Deliberately not registered with LocalDb: it is a per-node fact (which names this node mirrored), not site-wide state, and replicating it would make one node's mirroring progress masquerade as the other's.
|
||||
|
||||
**`DeployedSecretsMirror`** (`SiteRuntime/Secrets/DeployedSecretsMirror.cs`) is a per-node `BackgroundService` — registered on **every** site node, not a singleton — that reconciles `deployed_secrets` into the node's own local `ISecretStore` (site nodes never run a replication-decorated store; see Component-Host.md → *Secrets*):
|
||||
@@ -449,6 +451,7 @@ Site secrets ride the artifact deploy (design `docs/plans/2026-09-28-site-secret
|
||||
- **Isolation.** One row failing to apply/verify does not block any other row in the same pass.
|
||||
- **Cache invalidation.** A successful apply or delete invalidates the shared `ISecretResolver` cache before verifying, so the verify step (and any script/connection reading the secret) sees the fresh value immediately rather than waiting out the resolver's cache TTL.
|
||||
- **Never logs a secret value** — log lines and the reported `MirrorResult` carry only names, revisions, and `KekId`s.
|
||||
- **Mirror status on the health report (review finding M1).** A failure on the **standby** — above all a secret that will not decrypt because the standby's master key differs from central's — used to live only in that node's log, invisible until a failover broke its connections. After every pass the mirror records its node's current failure set in `secret_mirror_status`, but **only when the set (sorted name+message pairs) differs from what this process last wrote**, plus once on the first pass after start-up so a restart refreshes a stale row — never on every 15 s tick, because every write is a replicated CDC change. The write is best-effort: a failure logs a Warning (repeats at Debug), never affects the pass result, and is retried next pass. `SecretMirrorStatusReporter` (`SiteRuntime/Secrets/`, every site node, 30 s, immediate first probe) reads every node's row into `ISiteHealthCollector.SetSecretMirrorStatuses`, so the active node's report carries both nodes' status as `SiteHealthReport.SecretMirrorStatuses` (see Component-HealthMonitoring.md). On an **unreplicated** pair only the active node's own row is visible — the same replication prerequisite deployed secrets already have.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -89,7 +89,14 @@ public static class DeployCommands
|
||||
/// </summary>
|
||||
internal static readonly TimeSpan BulkDeployTimeout = Timeout.InfiniteTimeSpan;
|
||||
|
||||
internal static int HandleSiteDeploymentResponse(string json, string format)
|
||||
/// <summary>
|
||||
/// Renders a deployment summary and turns its <c>failureCount</c> into the exit code:
|
||||
/// 0 when every site/instance succeeded, 1 when any failed. Shared by <c>deploy site</c>
|
||||
/// and both artifact-deploy commands (<c>deploy artifacts</c>, <c>site deploy-artifacts</c>),
|
||||
/// whose summaries all carry <c>failureCount</c> — a per-site failure is a successful HTTP
|
||||
/// call, so without this the process exited 0 and scripts could not detect it.
|
||||
/// </summary>
|
||||
internal static int HandleSiteDeploymentResponse(string json, string format, string summaryName = "site deployment")
|
||||
{
|
||||
CommandHelpers.HandleResponse(new ManagementResponse(200, json, null, null), format);
|
||||
try
|
||||
@@ -100,7 +107,7 @@ public static class DeployCommands
|
||||
return count == 0 ? 0 : 1;
|
||||
}
|
||||
catch (JsonException) { }
|
||||
OutputFormatter.WriteError("The server returned an invalid site deployment summary.", "INVALID_RESPONSE");
|
||||
OutputFormatter.WriteError($"The server returned an invalid {summaryName} summary.", "INVALID_RESPONSE");
|
||||
return 1;
|
||||
}
|
||||
|
||||
@@ -113,7 +120,9 @@ public static class DeployCommands
|
||||
{
|
||||
var siteId = result.GetValue(siteIdOption);
|
||||
return await CommandHelpers.ExecuteCommandAsync(
|
||||
result, urlOption, formatOption, usernameOption, passwordOption, new MgmtDeployArtifactsCommand(siteId));
|
||||
result, urlOption, formatOption, usernameOption, passwordOption, new MgmtDeployArtifactsCommand(siteId),
|
||||
onSuccess: json => HandleSiteDeploymentResponse(
|
||||
json, CommandHelpers.ResolveFormat(result, formatOption, CliConfig.Load()), "artifact deployment"));
|
||||
});
|
||||
return cmd;
|
||||
}
|
||||
|
||||
@@ -222,8 +222,11 @@ public static class SiteCommands
|
||||
cmd.SetAction(async (ParseResult result) =>
|
||||
{
|
||||
var siteId = result.GetValue(siteIdOption);
|
||||
// Exit 1 when any site failed (see DeployCommands.HandleSiteDeploymentResponse).
|
||||
return await CommandHelpers.ExecuteCommandAsync(
|
||||
result, urlOption, formatOption, usernameOption, passwordOption, new MgmtDeployArtifactsCommand(siteId));
|
||||
result, urlOption, formatOption, usernameOption, passwordOption, new MgmtDeployArtifactsCommand(siteId),
|
||||
onSuccess: json => DeployCommands.HandleSiteDeploymentResponse(
|
||||
json, CommandHelpers.ResolveFormat(result, formatOption, CliConfig.Load()), "artifact deployment"));
|
||||
});
|
||||
return cmd;
|
||||
}
|
||||
|
||||
@@ -965,6 +965,11 @@ scadabridge --url <url> site deploy-artifacts [--site-id <int>]
|
||||
|--------|----------|-------------|
|
||||
| `--site-id` | no | Target site ID; deploys system-wide artifacts to exactly that site. Omit to deploy fleet-wide to all sites. A site-scoped (non-Administrator) Deployer must supply an in-scope `--site-id`; fleet-wide deployment requires a system-wide Deployer or Administrator (arch-review C2). |
|
||||
|
||||
Output is the per-site result matrix (`siteId`, `siteName`, `success`, `errorMessage`,
|
||||
`secretsDeployed`, `secretsRemoved`) plus `successCount` / `failureCount`. The command exits `1`
|
||||
when `failureCount` is non-zero — a site that failed (unreachable, a missing or unreadable
|
||||
secret, a partial secret apply) is a failed command even though the HTTP call succeeded.
|
||||
|
||||
#### `site area list`
|
||||
|
||||
List all areas for a site.
|
||||
@@ -1107,6 +1112,8 @@ scadabridge --url <url> deploy artifacts [--site-id <int>]
|
||||
|--------|----------|-------------|
|
||||
| `--site-id` | no | Target site ID; deploys to exactly that site. Omit to deploy fleet-wide. A site-scoped (non-Administrator) Deployer must supply an in-scope `--site-id`; fleet-wide requires a system-wide Deployer or Administrator (arch-review C2). |
|
||||
|
||||
Same output and exit-code rule as `site deploy-artifacts`: exit `1` when any site failed.
|
||||
|
||||
#### `deploy status`
|
||||
|
||||
Query deployment records, with optional filters and pagination.
|
||||
|
||||
@@ -275,6 +275,14 @@
|
||||
No self-heal
|
||||
</span>
|
||||
}
|
||||
@if (HasSecretMirrorFailures(state))
|
||||
{
|
||||
<span class="badge bg-warning text-dark"
|
||||
data-test="secret-mirror-badge"
|
||||
title="A site node could not apply and/or decrypt one or more deployed secrets. A standby node's failures surface here before a failover would expose them — see the details in the card body.">
|
||||
Secret mirror failing
|
||||
</span>
|
||||
}
|
||||
@if (!state.IsOnline && state.LastStatusChangeAt is { } changedAt)
|
||||
{
|
||||
<small class="text-muted">offline since @changedAt.ToString("u")</small>
|
||||
@@ -302,6 +310,26 @@
|
||||
@if (state.LatestReport != null)
|
||||
{
|
||||
var report = state.LatestReport;
|
||||
@if (HasSecretMirrorFailures(state))
|
||||
{
|
||||
@* Review finding M1: every site node's deployed-secrets mirror failures,
|
||||
the standby's included (carried on the active node's report). *@
|
||||
<div class="alert alert-warning py-2 px-3 mb-3 small" data-test="secret-mirror-failures" role="alert">
|
||||
<strong>Deployed secrets failing on a site node</strong>
|
||||
<ul class="mb-0 ps-3">
|
||||
@foreach (var node in report.SecretMirrorStatuses!.Where(n => n.Failures.Count > 0))
|
||||
{
|
||||
@foreach (var failure in node.Failures)
|
||||
{
|
||||
<li>
|
||||
<span class="font-monospace">@node.NodeName</span> ·
|
||||
<span class="font-monospace">@failure.SecretName</span>: @failure.Message
|
||||
</li>
|
||||
}
|
||||
}
|
||||
</ul>
|
||||
</div>
|
||||
}
|
||||
<div class="row g-3">
|
||||
@* Each logical group below sits in its own card, with the group's
|
||||
head promoted to a real card-header, instead of h6 rules stacked
|
||||
@@ -590,6 +618,15 @@
|
||||
state.IsOnline
|
||||
&& state.LatestReport?.StartupReconcilerActive == false;
|
||||
|
||||
// Review finding M1: warn when any site node (the standby included) reports deployed
|
||||
// secrets it cannot apply/decrypt. Null ("no data": central self-report, pre-M1 site) and
|
||||
// nodes with an empty failure list are healthy. Offline sites are skipped like the other
|
||||
// warning badges — the Offline badge is the signal that matters there.
|
||||
private static bool HasSecretMirrorFailures(SiteHealthState state) =>
|
||||
state.IsOnline
|
||||
&& state.LatestReport?.SecretMirrorStatuses is { } statuses
|
||||
&& statuses.Any(n => n.Failures.Count > 0);
|
||||
|
||||
// Online central nodes, from the same ClusterNodes list the Nodes column renders.
|
||||
// Fewer than two means no standby, which disables the manual-failover control. This is
|
||||
// a display-side guard only — AkkaManualFailoverService re-checks against live cluster
|
||||
|
||||
@@ -1,9 +1,22 @@
|
||||
namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
|
||||
|
||||
/// <summary>
|
||||
/// A site's reply to a <see cref="DeployArtifactsCommand"/>.
|
||||
/// </summary>
|
||||
/// <param name="SecretsApplied">
|
||||
/// How many of the shipped secrets this node applied and verified; <c>null</c> from a site that
|
||||
/// predates secret deployment or when central shipped no secret list.
|
||||
/// </param>
|
||||
/// <param name="SecretsRemoved">
|
||||
/// Names of previously-deployed secrets this deploy retired (marked removed in the site's
|
||||
/// <c>deployed_secrets</c> table because the shipped set no longer contains them). Names only —
|
||||
/// never values. <c>null</c> when none were removed or the site predates the field.
|
||||
/// </param>
|
||||
public record ArtifactDeploymentResponse(
|
||||
string DeploymentId,
|
||||
string SiteId,
|
||||
bool Success,
|
||||
string? ErrorMessage,
|
||||
DateTimeOffset Timestamp,
|
||||
int? SecretsApplied = null);
|
||||
int? SecretsApplied = null,
|
||||
IReadOnlyList<string>? SecretsRemoved = null);
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
|
||||
|
||||
/// <summary>
|
||||
/// One site node's latest deployed-secrets mirror outcome, as carried on
|
||||
/// <see cref="SiteHealthReport.SecretMirrorStatuses"/> (review finding M1).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Every site node runs its own <c>DeployedSecretsMirror</c> and records its current
|
||||
/// failure set in the LocalDb-replicated <c>secret_mirror_status</c> table, keyed by node
|
||||
/// name; the ACTIVE node reads every row and ships them on its health report, so a
|
||||
/// STANDBY node's failures (most importantly a master-key mismatch that stops a deployed
|
||||
/// secret decrypting there) reach central before a failover exposes them. Carries names
|
||||
/// and human-readable messages only — never a secret value or ciphertext.
|
||||
/// </remarks>
|
||||
/// <param name="NodeName">The reporting node's semantic name (<c>node-a</c>/<c>node-b</c>), or its host name when unconfigured.</param>
|
||||
/// <param name="CheckedAtUtc">When that node last recorded a CHANGE in its failure set (or its first pass after start-up) — not the time of its latest pass.</param>
|
||||
/// <param name="Failures">That node's current mirror failures; empty when the node's mirror is healthy.</param>
|
||||
public sealed record SecretMirrorNodeStatus(
|
||||
string NodeName,
|
||||
DateTimeOffset CheckedAtUtc,
|
||||
IReadOnlyList<SecretMirrorFailure> Failures);
|
||||
|
||||
/// <summary>A single deployed secret that failed to apply and/or decrypt on one site node.</summary>
|
||||
/// <param name="SecretName">The secret's name.</param>
|
||||
/// <param name="Message">A human-readable failure message — never includes the secret's value.</param>
|
||||
public sealed record SecretMirrorFailure(string SecretName, string Message);
|
||||
@@ -173,6 +173,20 @@ public record SiteHealthReport(
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public bool? StartupReconcilerActive { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Every site node's latest deployed-secrets mirror outcome (review finding M1) — one
|
||||
/// entry per node that has recorded a row in the replicated <c>secret_mirror_status</c>
|
||||
/// table, including the STANDBY node, whose own health report is never sent.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <b>Null = no data</b>: a central self-report, a pre-M1 site build, a site whose
|
||||
/// reporter has not run yet, or a failed read. An EMPTY list means the feature is wired
|
||||
/// but no node has recorded a status yet. A node entry with an empty
|
||||
/// <see cref="SecretMirrorNodeStatus.Failures"/> list is a healthy mirror. Names and
|
||||
/// messages only — never secret values or ciphertext.
|
||||
/// </remarks>
|
||||
public IReadOnlyList<SecretMirrorNodeStatus>? SecretMirrorStatuses { get; init; }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -559,6 +559,32 @@ public static class CentralControlDtoMapper
|
||||
dto.SiteAuditBacklog = snapshot;
|
||||
}
|
||||
|
||||
// Review finding M1: wrapper presence keeps null ("no data") distinct from empty.
|
||||
if (msg.SecretMirrorStatuses is { } mirrorStatuses)
|
||||
{
|
||||
var wrapper = new SecretMirrorStatusListDto();
|
||||
foreach (var node in mirrorStatuses)
|
||||
{
|
||||
var nodeDto = new SecretMirrorNodeStatusDto
|
||||
{
|
||||
NodeName = node.NodeName,
|
||||
CheckedAtUtc = Timestamp.FromDateTimeOffset(node.CheckedAtUtc),
|
||||
};
|
||||
foreach (var failure in node.Failures)
|
||||
{
|
||||
nodeDto.Failures.Add(new SecretMirrorFailureDto
|
||||
{
|
||||
SecretName = failure.SecretName,
|
||||
Message = failure.Message,
|
||||
});
|
||||
}
|
||||
|
||||
wrapper.Nodes.Add(nodeDto);
|
||||
}
|
||||
|
||||
dto.SecretMirrorStatuses = wrapper;
|
||||
}
|
||||
|
||||
return dto;
|
||||
}
|
||||
|
||||
@@ -620,6 +646,19 @@ public static class CentralControlDtoMapper
|
||||
OnDiskBytes: snapshot.OnDiskBytes);
|
||||
}
|
||||
|
||||
List<SecretMirrorNodeStatus>? mirrorStatuses = null;
|
||||
if (dto.SecretMirrorStatuses is { } mirrorWrapper)
|
||||
{
|
||||
mirrorStatuses = new List<SecretMirrorNodeStatus>(mirrorWrapper.Nodes.Count);
|
||||
foreach (var node in mirrorWrapper.Nodes)
|
||||
{
|
||||
mirrorStatuses.Add(new SecretMirrorNodeStatus(
|
||||
node.NodeName,
|
||||
node.CheckedAtUtc?.ToDateTimeOffset() ?? DateTimeOffset.MinValue,
|
||||
node.Failures.Select(f => new SecretMirrorFailure(f.SecretName, f.Message)).ToList()));
|
||||
}
|
||||
}
|
||||
|
||||
return new SiteHealthReport(
|
||||
SiteId: dto.SiteId,
|
||||
SequenceNumber: dto.SequenceNumber,
|
||||
@@ -657,6 +696,7 @@ public static class CentralControlDtoMapper
|
||||
DetachedScriptThreads = dto.DetachedScriptThreads,
|
||||
ScriptRunShedCount = dto.ScriptRunShedCount,
|
||||
SiteStreamAlarmDropCount = dto.SiteStreamAlarmDropCount,
|
||||
SecretMirrorStatuses = mirrorStatuses,
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
@@ -827,6 +827,11 @@ public static class SiteCommandDtoMapper
|
||||
dto.SecretsApplied = response.SecretsApplied.Value;
|
||||
}
|
||||
|
||||
if (response.SecretsRemoved is { Count: > 0 })
|
||||
{
|
||||
dto.SecretsRemoved.AddRange(response.SecretsRemoved);
|
||||
}
|
||||
|
||||
return dto;
|
||||
}
|
||||
|
||||
@@ -839,7 +844,8 @@ public static class SiteCommandDtoMapper
|
||||
|
||||
return new ArtifactDeploymentResponse(
|
||||
dto.DeploymentId, dto.SiteId, dto.Success, NullIfEmpty(dto.ErrorMessage), Dto(dto.Timestamp),
|
||||
dto.SecretsApplied);
|
||||
dto.SecretsApplied,
|
||||
dto.SecretsRemoved.Count > 0 ? dto.SecretsRemoved.ToList() : null);
|
||||
}
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────
|
||||
|
||||
@@ -193,6 +193,25 @@ message NodeStatusListDto {
|
||||
repeated NodeStatusDto nodes = 1;
|
||||
}
|
||||
|
||||
// Review finding M1: one site node's deployed-secrets mirror failures. Names and
|
||||
// human-readable messages only — never a secret value or ciphertext.
|
||||
message SecretMirrorFailureDto {
|
||||
string secret_name = 1;
|
||||
string message = 2;
|
||||
}
|
||||
|
||||
message SecretMirrorNodeStatusDto {
|
||||
string node_name = 1;
|
||||
google.protobuf.Timestamp checked_at_utc = 2;
|
||||
repeated SecretMirrorFailureDto failures = 3;
|
||||
}
|
||||
|
||||
// Presence wrapper (same reason as NodeStatusListDto): absent = null ("no data"),
|
||||
// present-but-empty = the feature is wired and no node has recorded a status yet.
|
||||
message SecretMirrorStatusListDto {
|
||||
repeated SecretMirrorNodeStatusDto nodes = 1;
|
||||
}
|
||||
|
||||
// Site -> Central: periodic site health report. Mirrors Commons SiteHealthReport.
|
||||
// Additive-only evolution: field numbers are never reused.
|
||||
message SiteHealthReportDto {
|
||||
@@ -245,6 +264,10 @@ message SiteHealthReportDto {
|
||||
// dedicated alarm publish queue because it was at capacity. Point-in-time
|
||||
// (not reset on collect), like site_event_log_write_failures.
|
||||
int64 site_stream_alarm_drop_count = 32;
|
||||
// Review finding M1: every site node's deployed-secrets mirror status (the standby's
|
||||
// included — it is read from a LocalDb-replicated table on the active node).
|
||||
// Absent means null ("no data": central self-report, pre-M1 site, or not read yet).
|
||||
SecretMirrorStatusListDto secret_mirror_statuses = 33;
|
||||
}
|
||||
|
||||
// Central -> Site: health report ack, so a lost report is observable.
|
||||
|
||||
@@ -340,6 +340,7 @@ message ArtifactDeploymentResponseDto {
|
||||
string error_message = 4; // empty string represents null
|
||||
google.protobuf.Timestamp timestamp = 5;
|
||||
google.protobuf.Int32Value secrets_applied = 6; // absent => null
|
||||
repeated string secrets_removed = 7; // names only; empty => null
|
||||
}
|
||||
|
||||
message LifecycleRequest {
|
||||
|
||||
@@ -134,385 +134,385 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
"cmVsYXRpb25faWQYASABKAkSHAoUaW5zdGFuY2VfdW5pcXVlX25hbWUYAiAB",
|
||||
"KAkSEwoLaXNfZGVwbG95ZWQYAyABKAgSHQoVYXBwbGllZF9kZXBsb3ltZW50",
|
||||
"X2lkGAQgASgJEh0KFWFwcGxpZWRfcmV2aXNpb25faGFzaBgFIAEoCRItCgl0",
|
||||
"aW1lc3RhbXAYBiABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wItQB",
|
||||
"aW1lc3RhbXAYBiABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wIu0B",
|
||||
"Ch1BcnRpZmFjdERlcGxveW1lbnRSZXNwb25zZUR0bxIVCg1kZXBsb3ltZW50",
|
||||
"X2lkGAEgASgJEg8KB3NpdGVfaWQYAiABKAkSDwoHc3VjY2VzcxgDIAEoCBIV",
|
||||
"Cg1lcnJvcl9tZXNzYWdlGAQgASgJEi0KCXRpbWVzdGFtcBgFIAEoCzIaLmdv",
|
||||
"b2dsZS5wcm90b2J1Zi5UaW1lc3RhbXASNAoPc2VjcmV0c19hcHBsaWVkGAYg",
|
||||
"ASgLMhsuZ29vZ2xlLnByb3RvYnVmLkludDMyVmFsdWUimgQKEExpZmVjeWNs",
|
||||
"ZVJlcXVlc3QSVQoScmVmcmVzaF9kZXBsb3ltZW50GAEgASgLMjcuc2NhZGFi",
|
||||
"cmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUmVmcmVzaERlcGxveW1lbnRDb21tYW5k",
|
||||
"RHRvSAASTwoPZW5hYmxlX2luc3RhbmNlGAIgASgLMjQuc2NhZGFicmlkZ2Uu",
|
||||
"c2l0ZWNvbW1hbmQudjEuRW5hYmxlSW5zdGFuY2VDb21tYW5kRHRvSAASUQoQ",
|
||||
"ZGlzYWJsZV9pbnN0YW5jZRgDIAEoCzI1LnNjYWRhYnJpZGdlLnNpdGVjb21t",
|
||||
"YW5kLnYxLkRpc2FibGVJbnN0YW5jZUNvbW1hbmREdG9IABJPCg9kZWxldGVf",
|
||||
"aW5zdGFuY2UYBCABKAsyNC5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5E",
|
||||
"ZWxldGVJbnN0YW5jZUNvbW1hbmREdG9IABJcChZkZXBsb3ltZW50X3N0YXRl",
|
||||
"X3F1ZXJ5GAUgASgLMjouc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuRGVw",
|
||||
"bG95bWVudFN0YXRlUXVlcnlSZXF1ZXN0RHRvSAASUQoQZGVwbG95X2FydGlm",
|
||||
"YWN0cxgGIAEoCzI1LnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLkRlcGxv",
|
||||
"eUFydGlmYWN0c0NvbW1hbmREdG9IAEIJCgdjb21tYW5kIoADCg5MaWZlY3lj",
|
||||
"bGVSZXBseRJUChFkZXBsb3ltZW50X3N0YXR1cxgBIAEoCzI3LnNjYWRhYnJp",
|
||||
"ZGdlLnNpdGVjb21tYW5kLnYxLkRlcGxveW1lbnRTdGF0dXNSZXNwb25zZUR0",
|
||||
"b0gAElYKEmluc3RhbmNlX2xpZmVjeWNsZRgCIAEoCzI4LnNjYWRhYnJpZGdl",
|
||||
"LnNpdGVjb21tYW5kLnYxLkluc3RhbmNlTGlmZWN5Y2xlUmVzcG9uc2VEdG9I",
|
||||
"ABJdChZkZXBsb3ltZW50X3N0YXRlX3F1ZXJ5GAMgASgLMjsuc2NhZGFicmlk",
|
||||
"Z2Uuc2l0ZWNvbW1hbmQudjEuRGVwbG95bWVudFN0YXRlUXVlcnlSZXNwb25z",
|
||||
"ZUR0b0gAElgKE2FydGlmYWN0X2RlcGxveW1lbnQYBCABKAsyOS5zY2FkYWJy",
|
||||
"aWRnZS5zaXRlY29tbWFuZC52MS5BcnRpZmFjdERlcGxveW1lbnRSZXNwb25z",
|
||||
"ZUR0b0gAQgcKBXJlcGx5InwKFEJyb3dzZU5vZGVDb21tYW5kRHRvEhcKD2Nv",
|
||||
"bm5lY3Rpb25fbmFtZRgBIAEoCRIWCg5wYXJlbnRfbm9kZV9pZBgCIAEoCRIa",
|
||||
"ChJjb250aW51YXRpb25fdG9rZW4YAyABKAkSFwoPc2l0ZV9pZGVudGlmaWVy",
|
||||
"GAQgASgJIoICCg1Ccm93c2VOb2RlRHRvEg8KB25vZGVfaWQYASABKAkSFAoM",
|
||||
"ZGlzcGxheV9uYW1lGAIgASgJEkIKCm5vZGVfY2xhc3MYAyABKA4yLi5zY2Fk",
|
||||
"YWJyaWRnZS5zaXRlY29tbWFuZC52MS5Ccm93c2VOb2RlQ2xhc3NEdG8SFAoM",
|
||||
"aGFzX2NoaWxkcmVuGAQgASgIEhEKCWRhdGFfdHlwZRgFIAEoCRIvCgp2YWx1",
|
||||
"ZV9yYW5rGAYgASgLMhsuZ29vZ2xlLnByb3RvYnVmLkludDMyVmFsdWUSLAoI",
|
||||
"d3JpdGFibGUYByABKAsyGi5nb29nbGUucHJvdG9idWYuQm9vbFZhbHVlImMK",
|
||||
"EEJyb3dzZUZhaWx1cmVEdG8SPgoEa2luZBgBIAEoDjIwLnNjYWRhYnJpZGdl",
|
||||
"LnNpdGVjb21tYW5kLnYxLkJyb3dzZUZhaWx1cmVLaW5kRHRvEg8KB21lc3Nh",
|
||||
"Z2UYAiABKAkiwAEKE0Jyb3dzZU5vZGVSZXN1bHREdG8SOwoIY2hpbGRyZW4Y",
|
||||
"ASADKAsyKS5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5Ccm93c2VOb2Rl",
|
||||
"RHRvEhEKCXRydW5jYXRlZBgCIAEoCBI9CgdmYWlsdXJlGAMgASgLMiwuc2Nh",
|
||||
"ZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuQnJvd3NlRmFpbHVyZUR0bxIaChJj",
|
||||
"b250aW51YXRpb25fdG9rZW4YBCABKAkihwEKHFNlYXJjaEFkZHJlc3NTcGFj",
|
||||
"ZUNvbW1hbmREdG8SFwoPY29ubmVjdGlvbl9uYW1lGAEgASgJEg0KBXF1ZXJ5",
|
||||
"GAIgASgJEhEKCW1heF9kZXB0aBgDIAEoBRITCgttYXhfcmVzdWx0cxgEIAEo",
|
||||
"BRIXCg9zaXRlX2lkZW50aWZpZXIYBSABKAkiXQoUQWRkcmVzc1NwYWNlTWF0",
|
||||
"Y2hEdG8SNwoEbm9kZRgBIAEoCzIpLnNjYWRhYnJpZGdlLnNpdGVjb21tYW5k",
|
||||
"LnYxLkJyb3dzZU5vZGVEdG8SDAoEcGF0aBgCIAEoCSK0AQobU2VhcmNoQWRk",
|
||||
"cmVzc1NwYWNlUmVzdWx0RHRvEkEKB21hdGNoZXMYASADKAsyMC5zY2FkYWJy",
|
||||
"aWRnZS5zaXRlY29tbWFuZC52MS5BZGRyZXNzU3BhY2VNYXRjaER0bxITCgtj",
|
||||
"YXBfcmVhY2hlZBgCIAEoCBI9CgdmYWlsdXJlGAMgASgLMiwuc2NhZGFicmlk",
|
||||
"Z2Uuc2l0ZWNvbW1hbmQudjEuQnJvd3NlRmFpbHVyZUR0byJFChdSZWFkVGFn",
|
||||
"VmFsdWVzQ29tbWFuZER0bxIXCg9jb25uZWN0aW9uX25hbWUYASABKAkSEQoJ",
|
||||
"dGFnX3BhdGhzGAIgAygJIsQBChFUYWdSZWFkT3V0Y29tZUR0bxIQCgh0YWdf",
|
||||
"cGF0aBgBIAEoCRIPCgdzdWNjZXNzGAIgASgIEjUKBXZhbHVlGAMgASgLMiYu",
|
||||
"c2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuTG9vc2VWYWx1ZRIPCgdxdWFs",
|
||||
"aXR5GAQgASgJEi0KCXRpbWVzdGFtcBgFIAEoCzIaLmdvb2dsZS5wcm90b2J1",
|
||||
"Zi5UaW1lc3RhbXASFQoNZXJyb3JfbWVzc2FnZRgGIAEoCSJxChdSZWFkVGFn",
|
||||
"VmFsdWVzRmFpbHVyZUR0bxJFCgRraW5kGAEgASgOMjcuc2NhZGFicmlkZ2Uu",
|
||||
"c2l0ZWNvbW1hbmQudjEuUmVhZFRhZ1ZhbHVlc0ZhaWx1cmVLaW5kRHRvEg8K",
|
||||
"B21lc3NhZ2UYAiABKAkinwEKFlJlYWRUYWdWYWx1ZXNSZXN1bHREdG8SPwoI",
|
||||
"b3V0Y29tZXMYASADKAsyLS5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5U",
|
||||
"YWdSZWFkT3V0Y29tZUR0bxJECgdmYWlsdXJlGAIgASgLMjMuc2NhZGFicmlk",
|
||||
"Z2Uuc2l0ZWNvbW1hbmQudjEuUmVhZFRhZ1ZhbHVlc0ZhaWx1cmVEdG8icwoY",
|
||||
"VmVyaWZ5RW5kcG9pbnRDb21tYW5kRHRvEhcKD2Nvbm5lY3Rpb25fbmFtZRgB",
|
||||
"IAEoCRIQCghwcm90b2NvbBgCIAEoCRITCgtjb25maWdfanNvbhgDIAEoCRIX",
|
||||
"Cg9zaXRlX2lkZW50aWZpZXIYBCABKAkiwwEKEVNlcnZlckNlcnRJbmZvRHRv",
|
||||
"EhIKCnRodW1icHJpbnQYASABKAkSDwoHc3ViamVjdBgCIAEoCRIOCgZpc3N1",
|
||||
"ZXIYAyABKAkSMgoObm90X2JlZm9yZV91dGMYBCABKAsyGi5nb29nbGUucHJv",
|
||||
"dG9idWYuVGltZXN0YW1wEjEKDW5vdF9hZnRlcl91dGMYBSABKAsyGi5nb29n",
|
||||
"bGUucHJvdG9idWYuVGltZXN0YW1wEhIKCmRlcl9iYXNlNjQYBiABKAkiWQoW",
|
||||
"VmVyaWZ5RmFpbHVyZUtpbmRWYWx1ZRI/CgV2YWx1ZRgBIAEoDjIwLnNjYWRh",
|
||||
"YnJpZGdlLnNpdGVjb21tYW5kLnYxLlZlcmlmeUZhaWx1cmVLaW5kRHRvIsAB",
|
||||
"ChdWZXJpZnlFbmRwb2ludFJlc3VsdER0bxIPCgdzdWNjZXNzGAEgASgIEkgK",
|
||||
"DGZhaWx1cmVfa2luZBgCIAEoCzIyLnNjYWRhYnJpZGdlLnNpdGVjb21tYW5k",
|
||||
"LnYxLlZlcmlmeUZhaWx1cmVLaW5kVmFsdWUSDQoFZXJyb3IYAyABKAkSOwoE",
|
||||
"Y2VydBgEIAEoCzItLnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLlNlcnZl",
|
||||
"ckNlcnRJbmZvRHRvInUKGVRydXN0U2VydmVyQ2VydENvbW1hbmREdG8SFwoP",
|
||||
"Y29ubmVjdGlvbl9uYW1lGAEgASgJEhIKCmRlcl9iYXNlNjQYAiABKAkSEgoK",
|
||||
"dGh1bWJwcmludBgDIAEoCRIXCg9zaXRlX2lkZW50aWZpZXIYBCABKAkiNAoZ",
|
||||
"TGlzdFNlcnZlckNlcnRzQ29tbWFuZER0bxIXCg9zaXRlX2lkZW50aWZpZXIY",
|
||||
"ASABKAkiSQoaUmVtb3ZlU2VydmVyQ2VydENvbW1hbmREdG8SEgoKdGh1bWJw",
|
||||
"cmludBgBIAEoCRIXCg9zaXRlX2lkZW50aWZpZXIYAiABKAkiwgEKElRydXN0",
|
||||
"ZWRDZXJ0SW5mb0R0bxISCgp0aHVtYnByaW50GAEgASgJEg8KB3N1YmplY3QY",
|
||||
"AiABKAkSDgoGaXNzdWVyGAMgASgJEjIKDm5vdF9iZWZvcmVfdXRjGAQgASgL",
|
||||
"MhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcBIxCg1ub3RfYWZ0ZXJfdXRj",
|
||||
"GAUgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcBIQCghyZWplY3Rl",
|
||||
"ZBgGIAEoCCJXChZUcnVzdGVkQ2VydEluZm9MaXN0RHRvEj0KBWl0ZW1zGAEg",
|
||||
"AygLMi4uc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuVHJ1c3RlZENlcnRJ",
|
||||
"bmZvRHRvIncKEkNlcnRUcnVzdFJlc3VsdER0bxIPCgdzdWNjZXNzGAEgASgI",
|
||||
"Eg0KBWVycm9yGAIgASgJEkEKBWNlcnRzGAMgASgLMjIuc2NhZGFicmlkZ2Uu",
|
||||
"c2l0ZWNvbW1hbmQudjEuVHJ1c3RlZENlcnRJbmZvTGlzdER0byK9AQoSV3Jp",
|
||||
"dGVUYWdSZXF1ZXN0RHRvEhYKDmNvcnJlbGF0aW9uX2lkGAEgASgJEhcKD2Nv",
|
||||
"bm5lY3Rpb25fbmFtZRgCIAEoCRIQCgh0YWdfcGF0aBgDIAEoCRI1CgV2YWx1",
|
||||
"ZRgEIAEoCzImLnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLkxvb3NlVmFs",
|
||||
"dWUSLQoJdGltZXN0YW1wGAUgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVz",
|
||||
"dGFtcCKEAQoTV3JpdGVUYWdSZXNwb25zZUR0bxIWCg5jb3JyZWxhdGlvbl9p",
|
||||
"ZBgBIAEoCRIPCgdzdWNjZXNzGAIgASgIEhUKDWVycm9yX21lc3NhZ2UYAyAB",
|
||||
"KAkSLQoJdGltZXN0YW1wGAQgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVz",
|
||||
"dGFtcCKgBQoMT3BjVWFSZXF1ZXN0EkcKC2Jyb3dzZV9ub2RlGAEgASgLMjAu",
|
||||
"c2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuQnJvd3NlTm9kZUNvbW1hbmRE",
|
||||
"dG9IABJYChRzZWFyY2hfYWRkcmVzc19zcGFjZRgCIAEoCzI4LnNjYWRhYnJp",
|
||||
"ZGdlLnNpdGVjb21tYW5kLnYxLlNlYXJjaEFkZHJlc3NTcGFjZUNvbW1hbmRE",
|
||||
"dG9IABJOCg9yZWFkX3RhZ192YWx1ZXMYAyABKAsyMy5zY2FkYWJyaWRnZS5z",
|
||||
"aXRlY29tbWFuZC52MS5SZWFkVGFnVmFsdWVzQ29tbWFuZER0b0gAEk8KD3Zl",
|
||||
"cmlmeV9lbmRwb2ludBgEIAEoCzI0LnNjYWRhYnJpZGdlLnNpdGVjb21tYW5k",
|
||||
"LnYxLlZlcmlmeUVuZHBvaW50Q29tbWFuZER0b0gAElIKEXRydXN0X3NlcnZl",
|
||||
"cl9jZXJ0GAUgASgLMjUuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuVHJ1",
|
||||
"c3RTZXJ2ZXJDZXJ0Q29tbWFuZER0b0gAElIKEWxpc3Rfc2VydmVyX2NlcnRz",
|
||||
"GAYgASgLMjUuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuTGlzdFNlcnZl",
|
||||
"ckNlcnRzQ29tbWFuZER0b0gAElQKEnJlbW92ZV9zZXJ2ZXJfY2VydBgHIAEo",
|
||||
"CzI2LnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLlJlbW92ZVNlcnZlckNl",
|
||||
"cnRDb21tYW5kRHRvSAASQwoJd3JpdGVfdGFnGAggASgLMi4uc2NhZGFicmlk",
|
||||
"Z2Uuc2l0ZWNvbW1hbmQudjEuV3JpdGVUYWdSZXF1ZXN0RHRvSABCCQoHY29t",
|
||||
"bWFuZCLhAwoKT3BjVWFSZXBseRJGCgticm93c2Vfbm9kZRgBIAEoCzIvLnNj",
|
||||
"YWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLkJyb3dzZU5vZGVSZXN1bHREdG9I",
|
||||
"ABJXChRzZWFyY2hfYWRkcmVzc19zcGFjZRgCIAEoCzI3LnNjYWRhYnJpZGdl",
|
||||
"LnNpdGVjb21tYW5kLnYxLlNlYXJjaEFkZHJlc3NTcGFjZVJlc3VsdER0b0gA",
|
||||
"Ek0KD3JlYWRfdGFnX3ZhbHVlcxgDIAEoCzIyLnNjYWRhYnJpZGdlLnNpdGVj",
|
||||
"b21tYW5kLnYxLlJlYWRUYWdWYWx1ZXNSZXN1bHREdG9IABJOCg92ZXJpZnlf",
|
||||
"ZW5kcG9pbnQYBCABKAsyMy5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5W",
|
||||
"ZXJpZnlFbmRwb2ludFJlc3VsdER0b0gAEkQKCmNlcnRfdHJ1c3QYBSABKAsy",
|
||||
"Li5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5DZXJ0VHJ1c3RSZXN1bHRE",
|
||||
"dG9IABJECgl3cml0ZV90YWcYBiABKAsyLy5zY2FkYWJyaWRnZS5zaXRlY29t",
|
||||
"bWFuZC52MS5Xcml0ZVRhZ1Jlc3BvbnNlRHRvSABCBwoFcmVwbHkixQIKF0V2",
|
||||
"ZW50TG9nUXVlcnlSZXF1ZXN0RHRvEhYKDmNvcnJlbGF0aW9uX2lkGAEgASgJ",
|
||||
"Eg8KB3NpdGVfaWQYAiABKAkSKAoEZnJvbRgDIAEoCzIaLmdvb2dsZS5wcm90",
|
||||
"b2J1Zi5UaW1lc3RhbXASJgoCdG8YBCABKAsyGi5nb29nbGUucHJvdG9idWYu",
|
||||
"VGltZXN0YW1wEhIKCmV2ZW50X3R5cGUYBSABKAkSEAoIc2V2ZXJpdHkYBiAB",
|
||||
"KAkSEwoLaW5zdGFuY2VfaWQYByABKAkSFgoOa2V5d29yZF9maWx0ZXIYCCAB",
|
||||
"KAkSGgoSY29udGludWF0aW9uX3Rva2VuGAkgASgJEhEKCXBhZ2Vfc2l6ZRgK",
|
||||
"IAEoBRItCgl0aW1lc3RhbXAYCyABKAsyGi5nb29nbGUucHJvdG9idWYuVGlt",
|
||||
"ZXN0YW1wIroBChBFdmVudExvZ0VudHJ5RHRvEgoKAmlkGAEgASgJEi0KCXRp",
|
||||
"bWVzdGFtcBgCIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXASEgoK",
|
||||
"ZXZlbnRfdHlwZRgDIAEoCRIQCghzZXZlcml0eRgEIAEoCRITCgtpbnN0YW5j",
|
||||
"ZV9pZBgFIAEoCRIOCgZzb3VyY2UYBiABKAkSDwoHbWVzc2FnZRgHIAEoCRIP",
|
||||
"CgdkZXRhaWxzGAggASgJIocCChhFdmVudExvZ1F1ZXJ5UmVzcG9uc2VEdG8S",
|
||||
"FgoOY29ycmVsYXRpb25faWQYASABKAkSDwoHc2l0ZV9pZBgCIAEoCRI9Cgdl",
|
||||
"bnRyaWVzGAMgAygLMiwuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuRXZl",
|
||||
"bnRMb2dFbnRyeUR0bxIaChJjb250aW51YXRpb25fdG9rZW4YBCABKAkSEAoI",
|
||||
"aGFzX21vcmUYBSABKAgSDwoHc3VjY2VzcxgGIAEoCBIVCg1lcnJvcl9tZXNz",
|
||||
"YWdlGAcgASgJEi0KCXRpbWVzdGFtcBgIIAEoCzIaLmdvb2dsZS5wcm90b2J1",
|
||||
"Zi5UaW1lc3RhbXAiZAoXRGVidWdTbmFwc2hvdFJlcXVlc3REdG8SHAoUaW5z",
|
||||
"dGFuY2VfdW5pcXVlX25hbWUYASABKAkSFgoOY29ycmVsYXRpb25faWQYAiAB",
|
||||
"KAkSEwoLYWxhcm1zX29ubHkYAyABKAgiVAocU3Vic2NyaWJlRGVidWdWaWV3",
|
||||
"UmVxdWVzdER0bxIcChRpbnN0YW5jZV91bmlxdWVfbmFtZRgBIAEoCRIWCg5j",
|
||||
"b3JyZWxhdGlvbl9pZBgCIAEoCSJWCh5VbnN1YnNjcmliZURlYnVnVmlld1Jl",
|
||||
"cXVlc3REdG8SHAoUaW5zdGFuY2VfdW5pcXVlX25hbWUYASABKAkSFgoOY29y",
|
||||
"cmVsYXRpb25faWQYAiABKAkiHAoaVW5zdWJzY3JpYmVEZWJ1Z1ZpZXdBY2tE",
|
||||
"dG8i1AEKFkFsYXJtQ29uZGl0aW9uU3RhdGVEdG8SDgoGYWN0aXZlGAEgASgI",
|
||||
"EhQKDGFja25vd2xlZGdlZBgCIAEoCBItCgljb25maXJtZWQYAyABKAsyGi5n",
|
||||
"b29nbGUucHJvdG9idWYuQm9vbFZhbHVlEj8KBnNoZWx2ZRgEIAEoDjIvLnNj",
|
||||
"YWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLkFsYXJtU2hlbHZlU3RhdGVEdG8S",
|
||||
"EgoKc3VwcHJlc3NlZBgFIAEoCBIQCghzZXZlcml0eRgGIAEoBSLdAQoWRGVi",
|
||||
"dWdBdHRyaWJ1dGVWYWx1ZUR0bxIcChRpbnN0YW5jZV91bmlxdWVfbmFtZRgB",
|
||||
"IAEoCRIWCg5hdHRyaWJ1dGVfcGF0aBgCIAEoCRIWCg5hdHRyaWJ1dGVfbmFt",
|
||||
"ZRgDIAEoCRI1CgV2YWx1ZRgEIAEoCzImLnNjYWRhYnJpZGdlLnNpdGVjb21t",
|
||||
"YW5kLnYxLkxvb3NlVmFsdWUSDwoHcXVhbGl0eRgFIAEoCRItCgl0aW1lc3Rh",
|
||||
"bXAYBiABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wIq8FChJEZWJ1",
|
||||
"Z0FsYXJtU3RhdGVEdG8SHAoUaW5zdGFuY2VfdW5pcXVlX25hbWUYASABKAkS",
|
||||
"EgoKYWxhcm1fbmFtZRgCIAEoCRI4CgVzdGF0ZRgDIAEoDjIpLnNjYWRhYnJp",
|
||||
"ZGdlLnNpdGVjb21tYW5kLnYxLkFsYXJtU3RhdGVEdG8SEAoIcHJpb3JpdHkY",
|
||||
"BCABKAUSLQoJdGltZXN0YW1wGAUgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRp",
|
||||
"bWVzdGFtcBI4CgVsZXZlbBgGIAEoDjIpLnNjYWRhYnJpZGdlLnNpdGVjb21t",
|
||||
"YW5kLnYxLkFsYXJtTGV2ZWxEdG8SDwoHbWVzc2FnZRgHIAEoCRI2CgRraW5k",
|
||||
"GAggASgOMiguc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuQWxhcm1LaW5k",
|
||||
"RHRvEkUKCWNvbmRpdGlvbhgJIAEoCzIyLnNjYWRhYnJpZGdlLnNpdGVjb21t",
|
||||
"YW5kLnYxLkFsYXJtQ29uZGl0aW9uU3RhdGVEdG8SGAoQc291cmNlX3JlZmVy",
|
||||
"ZW5jZRgKIAEoCRIXCg9hbGFybV90eXBlX25hbWUYCyABKAkSEAoIY2F0ZWdv",
|
||||
"cnkYDCABKAkSFQoNb3BlcmF0b3JfdXNlchgNIAEoCRIYChBvcGVyYXRvcl9j",
|
||||
"b21tZW50GA4gASgJEjcKE29yaWdpbmFsX3JhaXNlX3RpbWUYDyABKAsyGi5n",
|
||||
"b29nbGUucHJvdG9idWYuVGltZXN0YW1wEhUKDWN1cnJlbnRfdmFsdWUYECAB",
|
||||
"KAkSEwoLbGltaXRfdmFsdWUYESABKAkSJAocbmF0aXZlX3NvdXJjZV9jYW5v",
|
||||
"bmljYWxfbmFtZRgSIAEoCRIhChlpc19jb25maWd1cmVkX3BsYWNlaG9sZGVy",
|
||||
"GBMgASgIIpwCChREZWJ1Z1ZpZXdTbmFwc2hvdER0bxIcChRpbnN0YW5jZV91",
|
||||
"bmlxdWVfbmFtZRgBIAEoCRJMChBhdHRyaWJ1dGVfdmFsdWVzGAIgAygLMjIu",
|
||||
"c2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuRGVidWdBdHRyaWJ1dGVWYWx1",
|
||||
"ZUR0bxJECgxhbGFybV9zdGF0ZXMYAyADKAsyLi5zY2FkYWJyaWRnZS5zaXRl",
|
||||
"Y29tbWFuZC52MS5EZWJ1Z0FsYXJtU3RhdGVEdG8SNgoSc25hcHNob3RfdGlt",
|
||||
"ZXN0YW1wGAQgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcBIaChJp",
|
||||
"bnN0YW5jZV9ub3RfZm91bmQYBSABKAgi8AIKDFF1ZXJ5UmVxdWVzdBJOCg9l",
|
||||
"dmVudF9sb2dfcXVlcnkYASABKAsyMy5zY2FkYWJyaWRnZS5zaXRlY29tbWFu",
|
||||
"ZC52MS5FdmVudExvZ1F1ZXJ5UmVxdWVzdER0b0gAEk0KDmRlYnVnX3NuYXBz",
|
||||
"aG90GAIgASgLMjMuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuRGVidWdT",
|
||||
"bmFwc2hvdFJlcXVlc3REdG9IABJYChRzdWJzY3JpYmVfZGVidWdfdmlldxgD",
|
||||
"IAEoCzI4LnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLlN1YnNjcmliZURl",
|
||||
"YnVnVmlld1JlcXVlc3REdG9IABJcChZ1bnN1YnNjcmliZV9kZWJ1Z192aWV3",
|
||||
"GAQgASgLMjouc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuVW5zdWJzY3Jp",
|
||||
"YmVEZWJ1Z1ZpZXdSZXF1ZXN0RHRvSABCCQoHY29tbWFuZCKRAgoKUXVlcnlS",
|
||||
"ZXBseRJPCg9ldmVudF9sb2dfcXVlcnkYASABKAsyNC5zY2FkYWJyaWRnZS5z",
|
||||
"aXRlY29tbWFuZC52MS5FdmVudExvZ1F1ZXJ5UmVzcG9uc2VEdG9IABJPChNk",
|
||||
"ZWJ1Z192aWV3X3NuYXBzaG90GAIgASgLMjAuc2NhZGFicmlkZ2Uuc2l0ZWNv",
|
||||
"bW1hbmQudjEuRGVidWdWaWV3U25hcHNob3REdG9IABJYChZ1bnN1YnNjcmli",
|
||||
"ZV9kZWJ1Z192aWV3GAMgASgLMjYuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQu",
|
||||
"djEuVW5zdWJzY3JpYmVEZWJ1Z1ZpZXdBY2tEdG9IAEIHCgVyZXBseSKeAQoc",
|
||||
"UGFya2VkTWVzc2FnZVF1ZXJ5UmVxdWVzdER0bxIWCg5jb3JyZWxhdGlvbl9p",
|
||||
"ZBgBIAEoCRIPCgdzaXRlX2lkGAIgASgJEhMKC3BhZ2VfbnVtYmVyGAMgASgF",
|
||||
"EhEKCXBhZ2Vfc2l6ZRgEIAEoBRItCgl0aW1lc3RhbXAYBSABKAsyGi5nb29n",
|
||||
"bGUucHJvdG9idWYuVGltZXN0YW1wIvICChVQYXJrZWRNZXNzYWdlRW50cnlE",
|
||||
"dG8SEgoKbWVzc2FnZV9pZBgBIAEoCRIVCg10YXJnZXRfc3lzdGVtGAIgASgJ",
|
||||
"EhMKC21ldGhvZF9uYW1lGAMgASgJEhUKDWVycm9yX21lc3NhZ2UYBCABKAkS",
|
||||
"FQoNYXR0ZW1wdF9jb3VudBgFIAEoBRI2ChJvcmlnaW5hbF90aW1lc3RhbXAY",
|
||||
"BiABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEjoKFmxhc3RfYXR0",
|
||||
"ZW1wdF90aW1lc3RhbXAYByABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0",
|
||||
"YW1wEhQKDG1heF9hdHRlbXB0cxgIIAEoBRJICghjYXRlZ29yeRgJIAEoDjI2",
|
||||
"LnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLlN0b3JlQW5kRm9yd2FyZENh",
|
||||
"dGVnb3J5RHRvEhcKD29yaWdpbl9pbnN0YW5jZRgKIAEoCSKhAgodUGFya2Vk",
|
||||
"TWVzc2FnZVF1ZXJ5UmVzcG9uc2VEdG8SFgoOY29ycmVsYXRpb25faWQYASAB",
|
||||
"KAkSDwoHc2l0ZV9pZBgCIAEoCRJDCghtZXNzYWdlcxgDIAMoCzIxLnNjYWRh",
|
||||
"YnJpZGdlLnNpdGVjb21tYW5kLnYxLlBhcmtlZE1lc3NhZ2VFbnRyeUR0bxIT",
|
||||
"Cgt0b3RhbF9jb3VudBgEIAEoBRITCgtwYWdlX251bWJlchgFIAEoBRIRCglw",
|
||||
"YWdlX3NpemUYBiABKAUSDwoHc3VjY2VzcxgHIAEoCBIVCg1lcnJvcl9tZXNz",
|
||||
"YWdlGAggASgJEi0KCXRpbWVzdGFtcBgJIAEoCzIaLmdvb2dsZS5wcm90b2J1",
|
||||
"Zi5UaW1lc3RhbXAiigEKHFBhcmtlZE1lc3NhZ2VSZXRyeVJlcXVlc3REdG8S",
|
||||
"FgoOY29ycmVsYXRpb25faWQYASABKAkSDwoHc2l0ZV9pZBgCIAEoCRISCgpt",
|
||||
"ZXNzYWdlX2lkGAMgASgJEi0KCXRpbWVzdGFtcBgEIAEoCzIaLmdvb2dsZS5w",
|
||||
"cm90b2J1Zi5UaW1lc3RhbXAiXwodUGFya2VkTWVzc2FnZVJldHJ5UmVzcG9u",
|
||||
"c2VEdG8SFgoOY29ycmVsYXRpb25faWQYASABKAkSDwoHc3VjY2VzcxgCIAEo",
|
||||
"CBIVCg1lcnJvcl9tZXNzYWdlGAMgASgJIowBCh5QYXJrZWRNZXNzYWdlRGlz",
|
||||
"Y2FyZFJlcXVlc3REdG8SFgoOY29ycmVsYXRpb25faWQYASABKAkSDwoHc2l0",
|
||||
"ZV9pZBgCIAEoCRISCgptZXNzYWdlX2lkGAMgASgJEi0KCXRpbWVzdGFtcBgE",
|
||||
"IAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXAiYQofUGFya2VkTWVz",
|
||||
"c2FnZURpc2NhcmRSZXNwb25zZUR0bxIWCg5jb3JyZWxhdGlvbl9pZBgBIAEo",
|
||||
"CRIPCgdzdWNjZXNzGAIgASgIEhUKDWVycm9yX21lc3NhZ2UYAyABKAkiTwoX",
|
||||
"UmV0cnlQYXJrZWRPcGVyYXRpb25EdG8SFgoOY29ycmVsYXRpb25faWQYASAB",
|
||||
"KAkSHAoUdHJhY2tlZF9vcGVyYXRpb25faWQYAiABKAkiUQoZRGlzY2FyZFBh",
|
||||
"cmtlZE9wZXJhdGlvbkR0bxIWCg5jb3JyZWxhdGlvbl9pZBgBIAEoCRIcChR0",
|
||||
"cmFja2VkX29wZXJhdGlvbl9pZBgCIAEoCSJdChtQYXJrZWRPcGVyYXRpb25B",
|
||||
"Y3Rpb25BY2tEdG8SFgoOY29ycmVsYXRpb25faWQYASABKAkSDwoHYXBwbGll",
|
||||
"ZBgCIAEoCBIVCg1lcnJvcl9tZXNzYWdlGAMgASgJIt4DCg1QYXJrZWRSZXF1",
|
||||
"ZXN0ElgKFHBhcmtlZF9tZXNzYWdlX3F1ZXJ5GAEgASgLMjguc2NhZGFicmlk",
|
||||
"Z2Uuc2l0ZWNvbW1hbmQudjEuUGFya2VkTWVzc2FnZVF1ZXJ5UmVxdWVzdER0",
|
||||
"b0gAElgKFHBhcmtlZF9tZXNzYWdlX3JldHJ5GAIgASgLMjguc2NhZGFicmlk",
|
||||
"Z2Uuc2l0ZWNvbW1hbmQudjEuUGFya2VkTWVzc2FnZVJldHJ5UmVxdWVzdER0",
|
||||
"b0gAElwKFnBhcmtlZF9tZXNzYWdlX2Rpc2NhcmQYAyABKAsyOi5zY2FkYWJy",
|
||||
"aWRnZS5zaXRlY29tbWFuZC52MS5QYXJrZWRNZXNzYWdlRGlzY2FyZFJlcXVl",
|
||||
"c3REdG9IABJVChZyZXRyeV9wYXJrZWRfb3BlcmF0aW9uGAQgASgLMjMuc2Nh",
|
||||
"ZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUmV0cnlQYXJrZWRPcGVyYXRpb25E",
|
||||
"dG9IABJZChhkaXNjYXJkX3BhcmtlZF9vcGVyYXRpb24YBSABKAsyNS5zY2Fk",
|
||||
"YWJyaWRnZS5zaXRlY29tbWFuZC52MS5EaXNjYXJkUGFya2VkT3BlcmF0aW9u",
|
||||
"RHRvSABCCQoHY29tbWFuZCKHAwoLUGFya2VkUmVwbHkSWQoUcGFya2VkX21l",
|
||||
"c3NhZ2VfcXVlcnkYASABKAsyOS5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52",
|
||||
"MS5QYXJrZWRNZXNzYWdlUXVlcnlSZXNwb25zZUR0b0gAElkKFHBhcmtlZF9t",
|
||||
"ZXNzYWdlX3JldHJ5GAIgASgLMjkuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQu",
|
||||
"djEuUGFya2VkTWVzc2FnZVJldHJ5UmVzcG9uc2VEdG9IABJdChZwYXJrZWRf",
|
||||
"bWVzc2FnZV9kaXNjYXJkGAMgASgLMjsuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1h",
|
||||
"bmQudjEuUGFya2VkTWVzc2FnZURpc2NhcmRSZXNwb25zZUR0b0gAEloKF3Bh",
|
||||
"cmtlZF9vcGVyYXRpb25fYWN0aW9uGAQgASgLMjcuc2NhZGFicmlkZ2Uuc2l0",
|
||||
"ZWNvbW1hbmQudjEuUGFya2VkT3BlcmF0aW9uQWN0aW9uQWNrRHRvSABCBwoF",
|
||||
"cmVwbHki/AEKFVJvdXRlVG9DYWxsUmVxdWVzdER0bxIWCg5jb3JyZWxhdGlv",
|
||||
"bl9pZBgBIAEoCRIcChRpbnN0YW5jZV91bmlxdWVfbmFtZRgCIAEoCRITCgtz",
|
||||
"Y3JpcHRfbmFtZRgDIAEoCRI9CgpwYXJhbWV0ZXJzGAQgASgLMikuc2NhZGFi",
|
||||
"cmlkZ2Uuc2l0ZWNvbW1hbmQudjEuTG9vc2VWYWx1ZU1hcBItCgl0aW1lc3Rh",
|
||||
"bXAYBSABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEhsKE3BhcmVu",
|
||||
"dF9leGVjdXRpb25faWQYBiABKAkSDQoFdHJhY2UYByABKAgikgIKFlJvdXRl",
|
||||
"VG9DYWxsUmVzcG9uc2VEdG8SFgoOY29ycmVsYXRpb25faWQYASABKAkSDwoH",
|
||||
"c3VjY2VzcxgCIAEoCBI8CgxyZXR1cm5fdmFsdWUYAyABKAsyJi5zY2FkYWJy",
|
||||
"aWRnZS5zaXRlY29tbWFuZC52MS5Mb29zZVZhbHVlEhUKDWVycm9yX21lc3Nh",
|
||||
"Z2UYBCABKAkSLQoJdGltZXN0YW1wGAUgASgLMhouZ29vZ2xlLnByb3RvYnVm",
|
||||
"LlRpbWVzdGFtcBIaChJjaGlsZF9leGVjdXRpb25faWQYBiABKAkSLwoMdHJh",
|
||||
"Y2VfZXZlbnRzGAcgAygLMhkuc2l0ZXN0cmVhbS5BdWRpdEV2ZW50RHRvIrsB",
|
||||
"Ch5Sb3V0ZVRvR2V0QXR0cmlidXRlc1JlcXVlc3REdG8SFgoOY29ycmVsYXRp",
|
||||
"b25faWQYASABKAkSHAoUaW5zdGFuY2VfdW5pcXVlX25hbWUYAiABKAkSFwoP",
|
||||
"YXR0cmlidXRlX25hbWVzGAMgAygJEi0KCXRpbWVzdGFtcBgEIAEoCzIaLmdv",
|
||||
"b2dsZS5wcm90b2J1Zi5UaW1lc3RhbXASGwoTcGFyZW50X2V4ZWN1dGlvbl9p",
|
||||
"ZBgFIAEoCSLLAQofUm91dGVUb0dldEF0dHJpYnV0ZXNSZXNwb25zZUR0bxIW",
|
||||
"Cg5jb3JyZWxhdGlvbl9pZBgBIAEoCRI5CgZ2YWx1ZXMYAiABKAsyKS5zY2Fk",
|
||||
"YWJyaWRnZS5zaXRlY29tbWFuZC52MS5Mb29zZVZhbHVlTWFwEg8KB3N1Y2Nl",
|
||||
"c3MYAyABKAgSFQoNZXJyb3JfbWVzc2FnZRgEIAEoCRItCgl0aW1lc3RhbXAY",
|
||||
"BSABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wItQCCh5Sb3V0ZVRv",
|
||||
"U2V0QXR0cmlidXRlc1JlcXVlc3REdG8SFgoOY29ycmVsYXRpb25faWQYASAB",
|
||||
"KAkSHAoUaW5zdGFuY2VfdW5pcXVlX25hbWUYAiABKAkSaQoQYXR0cmlidXRl",
|
||||
"X3ZhbHVlcxgDIAMoCzJPLnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLlJv",
|
||||
"dXRlVG9TZXRBdHRyaWJ1dGVzUmVxdWVzdER0by5BdHRyaWJ1dGVWYWx1ZXNF",
|
||||
"bnRyeRItCgl0aW1lc3RhbXAYBCABKAsyGi5nb29nbGUucHJvdG9idWYuVGlt",
|
||||
"ZXN0YW1wEhsKE3BhcmVudF9leGVjdXRpb25faWQYBSABKAkSDQoFdHJhY2UY",
|
||||
"BiABKAgaNgoUQXR0cmlidXRlVmFsdWVzRW50cnkSCwoDa2V5GAEgASgJEg0K",
|
||||
"BXZhbHVlGAIgASgJOgI4ASKQAQofUm91dGVUb1NldEF0dHJpYnV0ZXNSZXNw",
|
||||
"b25zZUR0bxIWCg5jb3JyZWxhdGlvbl9pZBgBIAEoCRIPCgdzdWNjZXNzGAIg",
|
||||
"ASgIEhUKDWVycm9yX21lc3NhZ2UYAyABKAkSLQoJdGltZXN0YW1wGAQgASgL",
|
||||
"MhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcCLDAgohUm91dGVUb1dhaXRG",
|
||||
"b3JBdHRyaWJ1dGVSZXF1ZXN0RHRvEhYKDmNvcnJlbGF0aW9uX2lkGAEgASgJ",
|
||||
"EhwKFGluc3RhbmNlX3VuaXF1ZV9uYW1lGAIgASgJEhYKDmF0dHJpYnV0ZV9u",
|
||||
"YW1lGAMgASgJEjoKFHRhcmdldF92YWx1ZV9lbmNvZGVkGAQgASgLMhwuZ29v",
|
||||
"Z2xlLnByb3RvYnVmLlN0cmluZ1ZhbHVlEioKB3RpbWVvdXQYBSABKAsyGS5n",
|
||||
"b29nbGUucHJvdG9idWYuRHVyYXRpb24SLQoJdGltZXN0YW1wGAYgASgLMhou",
|
||||
"Z29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcBIbChNwYXJlbnRfZXhlY3V0aW9u",
|
||||
"X2lkGAcgASgJEhwKFHJlcXVpcmVfZ29vZF9xdWFsaXR5GAggASgIIv8BCiJS",
|
||||
"b3V0ZVRvV2FpdEZvckF0dHJpYnV0ZVJlc3BvbnNlRHRvEhYKDmNvcnJlbGF0",
|
||||
"aW9uX2lkGAEgASgJEg8KB21hdGNoZWQYAiABKAgSNQoFdmFsdWUYAyABKAsy",
|
||||
"Ji5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5Mb29zZVZhbHVlEg8KB3F1",
|
||||
"YWxpdHkYBCABKAkSEQoJdGltZWRfb3V0GAUgASgIEg8KB3N1Y2Nlc3MYBiAB",
|
||||
"KAgSFQoNZXJyb3JfbWVzc2FnZRgHIAEoCRItCgl0aW1lc3RhbXAYCCABKAsy",
|
||||
"Gi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wIokDCgxSb3V0ZVJlcXVlc3QS",
|
||||
"SgoNcm91dGVfdG9fY2FsbBgBIAEoCzIxLnNjYWRhYnJpZGdlLnNpdGVjb21t",
|
||||
"YW5kLnYxLlJvdXRlVG9DYWxsUmVxdWVzdER0b0gAEl0KF3JvdXRlX3RvX2dl",
|
||||
"dF9hdHRyaWJ1dGVzGAIgASgLMjouc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQu",
|
||||
"djEuUm91dGVUb0dldEF0dHJpYnV0ZXNSZXF1ZXN0RHRvSAASXQoXcm91dGVf",
|
||||
"dG9fc2V0X2F0dHJpYnV0ZXMYAyABKAsyOi5zY2FkYWJyaWRnZS5zaXRlY29t",
|
||||
"bWFuZC52MS5Sb3V0ZVRvU2V0QXR0cmlidXRlc1JlcXVlc3REdG9IABJkChty",
|
||||
"b3V0ZV90b193YWl0X2Zvcl9hdHRyaWJ1dGUYBCABKAsyPS5zY2FkYWJyaWRn",
|
||||
"ZS5zaXRlY29tbWFuZC52MS5Sb3V0ZVRvV2FpdEZvckF0dHJpYnV0ZVJlcXVl",
|
||||
"c3REdG9IAEIJCgdjb21tYW5kIokDCgpSb3V0ZVJlcGx5EksKDXJvdXRlX3Rv",
|
||||
"X2NhbGwYASABKAsyMi5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5Sb3V0",
|
||||
"ZVRvQ2FsbFJlc3BvbnNlRHRvSAASXgoXcm91dGVfdG9fZ2V0X2F0dHJpYnV0",
|
||||
"ZXMYAiABKAsyOy5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5Sb3V0ZVRv",
|
||||
"R2V0QXR0cmlidXRlc1Jlc3BvbnNlRHRvSAASXgoXcm91dGVfdG9fc2V0X2F0",
|
||||
"dHJpYnV0ZXMYAyABKAsyOy5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5S",
|
||||
"b3V0ZVRvU2V0QXR0cmlidXRlc1Jlc3BvbnNlRHRvSAASZQobcm91dGVfdG9f",
|
||||
"d2FpdF9mb3JfYXR0cmlidXRlGAQgASgLMj4uc2NhZGFicmlkZ2Uuc2l0ZWNv",
|
||||
"bW1hbmQudjEuUm91dGVUb1dhaXRGb3JBdHRyaWJ1dGVSZXNwb25zZUR0b0gA",
|
||||
"QgcKBXJlcGx5IkEKFlRyaWdnZXJTaXRlRmFpbG92ZXJEdG8SFgoOY29ycmVs",
|
||||
"YXRpb25faWQYASABKAkSDwoHc2l0ZV9pZBgCIAEoCSJtChJTaXRlRmFpbG92",
|
||||
"ZXJBY2tEdG8SFgoOY29ycmVsYXRpb25faWQYASABKAkSEAoIYWNjZXB0ZWQY",
|
||||
"AiABKAgSFgoOdGFyZ2V0X2FkZHJlc3MYAyABKAkSFQoNZXJyb3JfbWVzc2Fn",
|
||||
"ZRgEIAEoCSrLAQoTRGVwbG95bWVudFN0YXR1c0R0bxIlCiFERVBMT1lNRU5U",
|
||||
"X1NUQVRVU19EVE9fVU5TUEVDSUZJRUQQABIhCh1ERVBMT1lNRU5UX1NUQVRV",
|
||||
"U19EVE9fUEVORElORxABEiUKIURFUExPWU1FTlRfU1RBVFVTX0RUT19JTl9Q",
|
||||
"Uk9HUkVTUxACEiEKHURFUExPWU1FTlRfU1RBVFVTX0RUT19TVUNDRVNTEAMS",
|
||||
"IAocREVQTE9ZTUVOVF9TVEFUVVNfRFRPX0ZBSUxFRBAEKsQBChJCcm93c2VO",
|
||||
"b2RlQ2xhc3NEdG8SJQohQlJPV1NFX05PREVfQ0xBU1NfRFRPX1VOU1BFQ0lG",
|
||||
"SUVEEAASIAocQlJPV1NFX05PREVfQ0xBU1NfRFRPX09CSkVDVBABEiIKHkJS",
|
||||
"T1dTRV9OT0RFX0NMQVNTX0RUT19WQVJJQUJMRRACEiAKHEJST1dTRV9OT0RF",
|
||||
"X0NMQVNTX0RUT19NRVRIT0QQAxIfChtCUk9XU0VfTk9ERV9DTEFTU19EVE9f",
|
||||
"T1RIRVIQBCqhAgoUQnJvd3NlRmFpbHVyZUtpbmREdG8SJwojQlJPV1NFX0ZB",
|
||||
"SUxVUkVfS0lORF9EVE9fVU5TUEVDSUZJRUQQABIwCixCUk9XU0VfRkFJTFVS",
|
||||
"RV9LSU5EX0RUT19DT05ORUNUSU9OX05PVF9GT1VORBABEjQKMEJST1dTRV9G",
|
||||
"QUlMVVJFX0tJTkRfRFRPX0NPTk5FQ1RJT05fTk9UX0NPTk5FQ1RFRBACEikK",
|
||||
"JUJST1dTRV9GQUlMVVJFX0tJTkRfRFRPX05PVF9CUk9XU0FCTEUQAxIjCh9C",
|
||||
"Uk9XU0VfRkFJTFVSRV9LSU5EX0RUT19USU1FT1VUEAQSKAokQlJPV1NFX0ZB",
|
||||
"SUxVUkVfS0lORF9EVE9fU0VSVkVSX0VSUk9SEAUqqgIKG1JlYWRUYWdWYWx1",
|
||||
"ZXNGYWlsdXJlS2luZER0bxIwCixSRUFEX1RBR19WQUxVRVNfRkFJTFVSRV9L",
|
||||
"SU5EX0RUT19VTlNQRUNJRklFRBAAEjkKNVJFQURfVEFHX1ZBTFVFU19GQUlM",
|
||||
"VVJFX0tJTkRfRFRPX0NPTk5FQ1RJT05fTk9UX0ZPVU5EEAESPQo5UkVBRF9U",
|
||||
"QUdfVkFMVUVTX0ZBSUxVUkVfS0lORF9EVE9fQ09OTkVDVElPTl9OT1RfQ09O",
|
||||
"TkVDVEVEEAISLAooUkVBRF9UQUdfVkFMVUVTX0ZBSUxVUkVfS0lORF9EVE9f",
|
||||
"VElNRU9VVBADEjEKLVJFQURfVEFHX1ZBTFVFU19GQUlMVVJFX0tJTkRfRFRP",
|
||||
"X1NFUlZFUl9FUlJPUhAEKpMCChRWZXJpZnlGYWlsdXJlS2luZER0bxInCiNW",
|
||||
"RVJJRllfRkFJTFVSRV9LSU5EX0RUT19VTlNQRUNJRklFRBAAEicKI1ZFUklG",
|
||||
"WV9GQUlMVVJFX0tJTkRfRFRPX1VOUkVBQ0hBQkxFEAESJwojVkVSSUZZX0ZB",
|
||||
"SUxVUkVfS0lORF9EVE9fQVVUSF9GQUlMRUQQAhIxCi1WRVJJRllfRkFJTFVS",
|
||||
"RV9LSU5EX0RUT19VTlRSVVNURURfQ0VSVElGSUNBVEUQAxIjCh9WRVJJRllf",
|
||||
"RkFJTFVSRV9LSU5EX0RUT19USU1FT1VUEAQSKAokVkVSSUZZX0ZBSUxVUkVf",
|
||||
"S0lORF9EVE9fU0VSVkVSX0VSUk9SEAUq5QEKGlN0b3JlQW5kRm9yd2FyZENh",
|
||||
"dGVnb3J5RHRvEi4KKlNUT1JFX0FORF9GT1JXQVJEX0NBVEVHT1JZX0RUT19V",
|
||||
"TlNQRUNJRklFRBAAEjIKLlNUT1JFX0FORF9GT1JXQVJEX0NBVEVHT1JZX0RU",
|
||||
"T19FWFRFUk5BTF9TWVNURU0QARIvCitTVE9SRV9BTkRfRk9SV0FSRF9DQVRF",
|
||||
"R09SWV9EVE9fTk9USUZJQ0FUSU9OEAISMgouU1RPUkVfQU5EX0ZPUldBUkRf",
|
||||
"Q0FURUdPUllfRFRPX0NBQ0hFRF9EQl9XUklURRADKmgKDUFsYXJtU3RhdGVE",
|
||||
"dG8SHwobQUxBUk1fU1RBVEVfRFRPX1VOU1BFQ0lGSUVEEAASGgoWQUxBUk1f",
|
||||
"U1RBVEVfRFRPX0FDVElWRRABEhoKFkFMQVJNX1NUQVRFX0RUT19OT1JNQUwQ",
|
||||
"Aiq5AQoNQWxhcm1MZXZlbER0bxIfChtBTEFSTV9MRVZFTF9EVE9fVU5TUEVD",
|
||||
"SUZJRUQQABIYChRBTEFSTV9MRVZFTF9EVE9fTk9ORRABEhcKE0FMQVJNX0xF",
|
||||
"VkVMX0RUT19MT1cQAhIbChdBTEFSTV9MRVZFTF9EVE9fTE9XX0xPVxADEhgK",
|
||||
"FEFMQVJNX0xFVkVMX0RUT19ISUdIEAQSHQoZQUxBUk1fTEVWRUxfRFRPX0hJ",
|
||||
"R0hfSElHSBAFKpIBCgxBbGFybUtpbmREdG8SHgoaQUxBUk1fS0lORF9EVE9f",
|
||||
"VU5TUEVDSUZJRUQQABIbChdBTEFSTV9LSU5EX0RUT19DT01QVVRFRBABEiAK",
|
||||
"HEFMQVJNX0tJTkRfRFRPX05BVElWRV9PUENfVUEQAhIjCh9BTEFSTV9LSU5E",
|
||||
"X0RUT19OQVRJVkVfTVhfQUNDRVNTEAMq6AEKE0FsYXJtU2hlbHZlU3RhdGVE",
|
||||
"dG8SJgoiQUxBUk1fU0hFTFZFX1NUQVRFX0RUT19VTlNQRUNJRklFRBAAEiQK",
|
||||
"IEFMQVJNX1NIRUxWRV9TVEFURV9EVE9fVU5TSEVMVkVEEAESKwonQUxBUk1f",
|
||||
"U0hFTFZFX1NUQVRFX0RUT19PTkVfU0hPVF9TSEVMVkVEEAISKAokQUxBUk1f",
|
||||
"U0hFTFZFX1NUQVRFX0RUT19USU1FRF9TSEVMVkVEEAMSLAooQUxBUk1fU0hF",
|
||||
"TFZFX1NUQVRFX0RUT19QRVJNQU5FTlRfU0hFTFZFRBAEMoQFChJTaXRlQ29t",
|
||||
"bWFuZFNlcnZpY2USbAoQRXhlY3V0ZUxpZmVjeWNsZRIsLnNjYWRhYnJpZGdl",
|
||||
"LnNpdGVjb21tYW5kLnYxLkxpZmVjeWNsZVJlcXVlc3QaKi5zY2FkYWJyaWRn",
|
||||
"ZS5zaXRlY29tbWFuZC52MS5MaWZlY3ljbGVSZXBseRJgCgxFeGVjdXRlT3Bj",
|
||||
"VWESKC5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5PcGNVYVJlcXVlc3Qa",
|
||||
"Ji5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5PcGNVYVJlcGx5EmAKDEV4",
|
||||
"ZWN1dGVRdWVyeRIoLnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLlF1ZXJ5",
|
||||
"UmVxdWVzdBomLnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLlF1ZXJ5UmVw",
|
||||
"bHkSYwoNRXhlY3V0ZVBhcmtlZBIpLnNjYWRhYnJpZGdlLnNpdGVjb21tYW5k",
|
||||
"LnYxLlBhcmtlZFJlcXVlc3QaJy5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52",
|
||||
"MS5QYXJrZWRSZXBseRJgCgxFeGVjdXRlUm91dGUSKC5zY2FkYWJyaWRnZS5z",
|
||||
"aXRlY29tbWFuZC52MS5Sb3V0ZVJlcXVlc3QaJi5zY2FkYWJyaWRnZS5zaXRl",
|
||||
"Y29tbWFuZC52MS5Sb3V0ZVJlcGx5EnUKD1RyaWdnZXJGYWlsb3ZlchIyLnNj",
|
||||
"YWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLlRyaWdnZXJTaXRlRmFpbG92ZXJE",
|
||||
"dG8aLi5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5TaXRlRmFpbG92ZXJB",
|
||||
"Y2tEdG9CK6oCKFpCLk1PTS5XVy5TY2FkYUJyaWRnZS5Db21tdW5pY2F0aW9u",
|
||||
"LkdycGNiBnByb3RvMw=="));
|
||||
"ASgLMhsuZ29vZ2xlLnByb3RvYnVmLkludDMyVmFsdWUSFwoPc2VjcmV0c19y",
|
||||
"ZW1vdmVkGAcgAygJIpoEChBMaWZlY3ljbGVSZXF1ZXN0ElUKEnJlZnJlc2hf",
|
||||
"ZGVwbG95bWVudBgBIAEoCzI3LnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYx",
|
||||
"LlJlZnJlc2hEZXBsb3ltZW50Q29tbWFuZER0b0gAEk8KD2VuYWJsZV9pbnN0",
|
||||
"YW5jZRgCIAEoCzI0LnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLkVuYWJs",
|
||||
"ZUluc3RhbmNlQ29tbWFuZER0b0gAElEKEGRpc2FibGVfaW5zdGFuY2UYAyAB",
|
||||
"KAsyNS5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5EaXNhYmxlSW5zdGFu",
|
||||
"Y2VDb21tYW5kRHRvSAASTwoPZGVsZXRlX2luc3RhbmNlGAQgASgLMjQuc2Nh",
|
||||
"ZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuRGVsZXRlSW5zdGFuY2VDb21tYW5k",
|
||||
"RHRvSAASXAoWZGVwbG95bWVudF9zdGF0ZV9xdWVyeRgFIAEoCzI6LnNjYWRh",
|
||||
"YnJpZGdlLnNpdGVjb21tYW5kLnYxLkRlcGxveW1lbnRTdGF0ZVF1ZXJ5UmVx",
|
||||
"dWVzdER0b0gAElEKEGRlcGxveV9hcnRpZmFjdHMYBiABKAsyNS5zY2FkYWJy",
|
||||
"aWRnZS5zaXRlY29tbWFuZC52MS5EZXBsb3lBcnRpZmFjdHNDb21tYW5kRHRv",
|
||||
"SABCCQoHY29tbWFuZCKAAwoOTGlmZWN5Y2xlUmVwbHkSVAoRZGVwbG95bWVu",
|
||||
"dF9zdGF0dXMYASABKAsyNy5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5E",
|
||||
"ZXBsb3ltZW50U3RhdHVzUmVzcG9uc2VEdG9IABJWChJpbnN0YW5jZV9saWZl",
|
||||
"Y3ljbGUYAiABKAsyOC5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5JbnN0",
|
||||
"YW5jZUxpZmVjeWNsZVJlc3BvbnNlRHRvSAASXQoWZGVwbG95bWVudF9zdGF0",
|
||||
"ZV9xdWVyeRgDIAEoCzI7LnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLkRl",
|
||||
"cGxveW1lbnRTdGF0ZVF1ZXJ5UmVzcG9uc2VEdG9IABJYChNhcnRpZmFjdF9k",
|
||||
"ZXBsb3ltZW50GAQgASgLMjkuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEu",
|
||||
"QXJ0aWZhY3REZXBsb3ltZW50UmVzcG9uc2VEdG9IAEIHCgVyZXBseSJ8ChRC",
|
||||
"cm93c2VOb2RlQ29tbWFuZER0bxIXCg9jb25uZWN0aW9uX25hbWUYASABKAkS",
|
||||
"FgoOcGFyZW50X25vZGVfaWQYAiABKAkSGgoSY29udGludWF0aW9uX3Rva2Vu",
|
||||
"GAMgASgJEhcKD3NpdGVfaWRlbnRpZmllchgEIAEoCSKCAgoNQnJvd3NlTm9k",
|
||||
"ZUR0bxIPCgdub2RlX2lkGAEgASgJEhQKDGRpc3BsYXlfbmFtZRgCIAEoCRJC",
|
||||
"Cgpub2RlX2NsYXNzGAMgASgOMi4uc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQu",
|
||||
"djEuQnJvd3NlTm9kZUNsYXNzRHRvEhQKDGhhc19jaGlsZHJlbhgEIAEoCBIR",
|
||||
"CglkYXRhX3R5cGUYBSABKAkSLwoKdmFsdWVfcmFuaxgGIAEoCzIbLmdvb2ds",
|
||||
"ZS5wcm90b2J1Zi5JbnQzMlZhbHVlEiwKCHdyaXRhYmxlGAcgASgLMhouZ29v",
|
||||
"Z2xlLnByb3RvYnVmLkJvb2xWYWx1ZSJjChBCcm93c2VGYWlsdXJlRHRvEj4K",
|
||||
"BGtpbmQYASABKA4yMC5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5Ccm93",
|
||||
"c2VGYWlsdXJlS2luZER0bxIPCgdtZXNzYWdlGAIgASgJIsABChNCcm93c2VO",
|
||||
"b2RlUmVzdWx0RHRvEjsKCGNoaWxkcmVuGAEgAygLMikuc2NhZGFicmlkZ2Uu",
|
||||
"c2l0ZWNvbW1hbmQudjEuQnJvd3NlTm9kZUR0bxIRCgl0cnVuY2F0ZWQYAiAB",
|
||||
"KAgSPQoHZmFpbHVyZRgDIAEoCzIsLnNjYWRhYnJpZGdlLnNpdGVjb21tYW5k",
|
||||
"LnYxLkJyb3dzZUZhaWx1cmVEdG8SGgoSY29udGludWF0aW9uX3Rva2VuGAQg",
|
||||
"ASgJIocBChxTZWFyY2hBZGRyZXNzU3BhY2VDb21tYW5kRHRvEhcKD2Nvbm5l",
|
||||
"Y3Rpb25fbmFtZRgBIAEoCRINCgVxdWVyeRgCIAEoCRIRCgltYXhfZGVwdGgY",
|
||||
"AyABKAUSEwoLbWF4X3Jlc3VsdHMYBCABKAUSFwoPc2l0ZV9pZGVudGlmaWVy",
|
||||
"GAUgASgJIl0KFEFkZHJlc3NTcGFjZU1hdGNoRHRvEjcKBG5vZGUYASABKAsy",
|
||||
"KS5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5Ccm93c2VOb2RlRHRvEgwK",
|
||||
"BHBhdGgYAiABKAkitAEKG1NlYXJjaEFkZHJlc3NTcGFjZVJlc3VsdER0bxJB",
|
||||
"CgdtYXRjaGVzGAEgAygLMjAuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEu",
|
||||
"QWRkcmVzc1NwYWNlTWF0Y2hEdG8SEwoLY2FwX3JlYWNoZWQYAiABKAgSPQoH",
|
||||
"ZmFpbHVyZRgDIAEoCzIsLnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLkJy",
|
||||
"b3dzZUZhaWx1cmVEdG8iRQoXUmVhZFRhZ1ZhbHVlc0NvbW1hbmREdG8SFwoP",
|
||||
"Y29ubmVjdGlvbl9uYW1lGAEgASgJEhEKCXRhZ19wYXRocxgCIAMoCSLEAQoR",
|
||||
"VGFnUmVhZE91dGNvbWVEdG8SEAoIdGFnX3BhdGgYASABKAkSDwoHc3VjY2Vz",
|
||||
"cxgCIAEoCBI1CgV2YWx1ZRgDIAEoCzImLnNjYWRhYnJpZGdlLnNpdGVjb21t",
|
||||
"YW5kLnYxLkxvb3NlVmFsdWUSDwoHcXVhbGl0eRgEIAEoCRItCgl0aW1lc3Rh",
|
||||
"bXAYBSABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEhUKDWVycm9y",
|
||||
"X21lc3NhZ2UYBiABKAkicQoXUmVhZFRhZ1ZhbHVlc0ZhaWx1cmVEdG8SRQoE",
|
||||
"a2luZBgBIAEoDjI3LnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLlJlYWRU",
|
||||
"YWdWYWx1ZXNGYWlsdXJlS2luZER0bxIPCgdtZXNzYWdlGAIgASgJIp8BChZS",
|
||||
"ZWFkVGFnVmFsdWVzUmVzdWx0RHRvEj8KCG91dGNvbWVzGAEgAygLMi0uc2Nh",
|
||||
"ZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuVGFnUmVhZE91dGNvbWVEdG8SRAoH",
|
||||
"ZmFpbHVyZRgCIAEoCzIzLnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLlJl",
|
||||
"YWRUYWdWYWx1ZXNGYWlsdXJlRHRvInMKGFZlcmlmeUVuZHBvaW50Q29tbWFu",
|
||||
"ZER0bxIXCg9jb25uZWN0aW9uX25hbWUYASABKAkSEAoIcHJvdG9jb2wYAiAB",
|
||||
"KAkSEwoLY29uZmlnX2pzb24YAyABKAkSFwoPc2l0ZV9pZGVudGlmaWVyGAQg",
|
||||
"ASgJIsMBChFTZXJ2ZXJDZXJ0SW5mb0R0bxISCgp0aHVtYnByaW50GAEgASgJ",
|
||||
"Eg8KB3N1YmplY3QYAiABKAkSDgoGaXNzdWVyGAMgASgJEjIKDm5vdF9iZWZv",
|
||||
"cmVfdXRjGAQgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcBIxCg1u",
|
||||
"b3RfYWZ0ZXJfdXRjGAUgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFt",
|
||||
"cBISCgpkZXJfYmFzZTY0GAYgASgJIlkKFlZlcmlmeUZhaWx1cmVLaW5kVmFs",
|
||||
"dWUSPwoFdmFsdWUYASABKA4yMC5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52",
|
||||
"MS5WZXJpZnlGYWlsdXJlS2luZER0byLAAQoXVmVyaWZ5RW5kcG9pbnRSZXN1",
|
||||
"bHREdG8SDwoHc3VjY2VzcxgBIAEoCBJICgxmYWlsdXJlX2tpbmQYAiABKAsy",
|
||||
"Mi5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5WZXJpZnlGYWlsdXJlS2lu",
|
||||
"ZFZhbHVlEg0KBWVycm9yGAMgASgJEjsKBGNlcnQYBCABKAsyLS5zY2FkYWJy",
|
||||
"aWRnZS5zaXRlY29tbWFuZC52MS5TZXJ2ZXJDZXJ0SW5mb0R0byJ1ChlUcnVz",
|
||||
"dFNlcnZlckNlcnRDb21tYW5kRHRvEhcKD2Nvbm5lY3Rpb25fbmFtZRgBIAEo",
|
||||
"CRISCgpkZXJfYmFzZTY0GAIgASgJEhIKCnRodW1icHJpbnQYAyABKAkSFwoP",
|
||||
"c2l0ZV9pZGVudGlmaWVyGAQgASgJIjQKGUxpc3RTZXJ2ZXJDZXJ0c0NvbW1h",
|
||||
"bmREdG8SFwoPc2l0ZV9pZGVudGlmaWVyGAEgASgJIkkKGlJlbW92ZVNlcnZl",
|
||||
"ckNlcnRDb21tYW5kRHRvEhIKCnRodW1icHJpbnQYASABKAkSFwoPc2l0ZV9p",
|
||||
"ZGVudGlmaWVyGAIgASgJIsIBChJUcnVzdGVkQ2VydEluZm9EdG8SEgoKdGh1",
|
||||
"bWJwcmludBgBIAEoCRIPCgdzdWJqZWN0GAIgASgJEg4KBmlzc3VlchgDIAEo",
|
||||
"CRIyCg5ub3RfYmVmb3JlX3V0YxgEIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5U",
|
||||
"aW1lc3RhbXASMQoNbm90X2FmdGVyX3V0YxgFIAEoCzIaLmdvb2dsZS5wcm90",
|
||||
"b2J1Zi5UaW1lc3RhbXASEAoIcmVqZWN0ZWQYBiABKAgiVwoWVHJ1c3RlZENl",
|
||||
"cnRJbmZvTGlzdER0bxI9CgVpdGVtcxgBIAMoCzIuLnNjYWRhYnJpZGdlLnNp",
|
||||
"dGVjb21tYW5kLnYxLlRydXN0ZWRDZXJ0SW5mb0R0byJ3ChJDZXJ0VHJ1c3RS",
|
||||
"ZXN1bHREdG8SDwoHc3VjY2VzcxgBIAEoCBINCgVlcnJvchgCIAEoCRJBCgVj",
|
||||
"ZXJ0cxgDIAEoCzIyLnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLlRydXN0",
|
||||
"ZWRDZXJ0SW5mb0xpc3REdG8ivQEKEldyaXRlVGFnUmVxdWVzdER0bxIWCg5j",
|
||||
"b3JyZWxhdGlvbl9pZBgBIAEoCRIXCg9jb25uZWN0aW9uX25hbWUYAiABKAkS",
|
||||
"EAoIdGFnX3BhdGgYAyABKAkSNQoFdmFsdWUYBCABKAsyJi5zY2FkYWJyaWRn",
|
||||
"ZS5zaXRlY29tbWFuZC52MS5Mb29zZVZhbHVlEi0KCXRpbWVzdGFtcBgFIAEo",
|
||||
"CzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXAihAEKE1dyaXRlVGFnUmVz",
|
||||
"cG9uc2VEdG8SFgoOY29ycmVsYXRpb25faWQYASABKAkSDwoHc3VjY2VzcxgC",
|
||||
"IAEoCBIVCg1lcnJvcl9tZXNzYWdlGAMgASgJEi0KCXRpbWVzdGFtcBgEIAEo",
|
||||
"CzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXAioAUKDE9wY1VhUmVxdWVz",
|
||||
"dBJHCgticm93c2Vfbm9kZRgBIAEoCzIwLnNjYWRhYnJpZGdlLnNpdGVjb21t",
|
||||
"YW5kLnYxLkJyb3dzZU5vZGVDb21tYW5kRHRvSAASWAoUc2VhcmNoX2FkZHJl",
|
||||
"c3Nfc3BhY2UYAiABKAsyOC5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5T",
|
||||
"ZWFyY2hBZGRyZXNzU3BhY2VDb21tYW5kRHRvSAASTgoPcmVhZF90YWdfdmFs",
|
||||
"dWVzGAMgASgLMjMuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUmVhZFRh",
|
||||
"Z1ZhbHVlc0NvbW1hbmREdG9IABJPCg92ZXJpZnlfZW5kcG9pbnQYBCABKAsy",
|
||||
"NC5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5WZXJpZnlFbmRwb2ludENv",
|
||||
"bW1hbmREdG9IABJSChF0cnVzdF9zZXJ2ZXJfY2VydBgFIAEoCzI1LnNjYWRh",
|
||||
"YnJpZGdlLnNpdGVjb21tYW5kLnYxLlRydXN0U2VydmVyQ2VydENvbW1hbmRE",
|
||||
"dG9IABJSChFsaXN0X3NlcnZlcl9jZXJ0cxgGIAEoCzI1LnNjYWRhYnJpZGdl",
|
||||
"LnNpdGVjb21tYW5kLnYxLkxpc3RTZXJ2ZXJDZXJ0c0NvbW1hbmREdG9IABJU",
|
||||
"ChJyZW1vdmVfc2VydmVyX2NlcnQYByABKAsyNi5zY2FkYWJyaWRnZS5zaXRl",
|
||||
"Y29tbWFuZC52MS5SZW1vdmVTZXJ2ZXJDZXJ0Q29tbWFuZER0b0gAEkMKCXdy",
|
||||
"aXRlX3RhZxgIIAEoCzIuLnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLldy",
|
||||
"aXRlVGFnUmVxdWVzdER0b0gAQgkKB2NvbW1hbmQi4QMKCk9wY1VhUmVwbHkS",
|
||||
"RgoLYnJvd3NlX25vZGUYASABKAsyLy5zY2FkYWJyaWRnZS5zaXRlY29tbWFu",
|
||||
"ZC52MS5Ccm93c2VOb2RlUmVzdWx0RHRvSAASVwoUc2VhcmNoX2FkZHJlc3Nf",
|
||||
"c3BhY2UYAiABKAsyNy5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5TZWFy",
|
||||
"Y2hBZGRyZXNzU3BhY2VSZXN1bHREdG9IABJNCg9yZWFkX3RhZ192YWx1ZXMY",
|
||||
"AyABKAsyMi5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5SZWFkVGFnVmFs",
|
||||
"dWVzUmVzdWx0RHRvSAASTgoPdmVyaWZ5X2VuZHBvaW50GAQgASgLMjMuc2Nh",
|
||||
"ZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuVmVyaWZ5RW5kcG9pbnRSZXN1bHRE",
|
||||
"dG9IABJECgpjZXJ0X3RydXN0GAUgASgLMi4uc2NhZGFicmlkZ2Uuc2l0ZWNv",
|
||||
"bW1hbmQudjEuQ2VydFRydXN0UmVzdWx0RHRvSAASRAoJd3JpdGVfdGFnGAYg",
|
||||
"ASgLMi8uc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuV3JpdGVUYWdSZXNw",
|
||||
"b25zZUR0b0gAQgcKBXJlcGx5IsUCChdFdmVudExvZ1F1ZXJ5UmVxdWVzdER0",
|
||||
"bxIWCg5jb3JyZWxhdGlvbl9pZBgBIAEoCRIPCgdzaXRlX2lkGAIgASgJEigK",
|
||||
"BGZyb20YAyABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEiYKAnRv",
|
||||
"GAQgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcBISCgpldmVudF90",
|
||||
"eXBlGAUgASgJEhAKCHNldmVyaXR5GAYgASgJEhMKC2luc3RhbmNlX2lkGAcg",
|
||||
"ASgJEhYKDmtleXdvcmRfZmlsdGVyGAggASgJEhoKEmNvbnRpbnVhdGlvbl90",
|
||||
"b2tlbhgJIAEoCRIRCglwYWdlX3NpemUYCiABKAUSLQoJdGltZXN0YW1wGAsg",
|
||||
"ASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcCK6AQoQRXZlbnRMb2dF",
|
||||
"bnRyeUR0bxIKCgJpZBgBIAEoCRItCgl0aW1lc3RhbXAYAiABKAsyGi5nb29n",
|
||||
"bGUucHJvdG9idWYuVGltZXN0YW1wEhIKCmV2ZW50X3R5cGUYAyABKAkSEAoI",
|
||||
"c2V2ZXJpdHkYBCABKAkSEwoLaW5zdGFuY2VfaWQYBSABKAkSDgoGc291cmNl",
|
||||
"GAYgASgJEg8KB21lc3NhZ2UYByABKAkSDwoHZGV0YWlscxgIIAEoCSKHAgoY",
|
||||
"RXZlbnRMb2dRdWVyeVJlc3BvbnNlRHRvEhYKDmNvcnJlbGF0aW9uX2lkGAEg",
|
||||
"ASgJEg8KB3NpdGVfaWQYAiABKAkSPQoHZW50cmllcxgDIAMoCzIsLnNjYWRh",
|
||||
"YnJpZGdlLnNpdGVjb21tYW5kLnYxLkV2ZW50TG9nRW50cnlEdG8SGgoSY29u",
|
||||
"dGludWF0aW9uX3Rva2VuGAQgASgJEhAKCGhhc19tb3JlGAUgASgIEg8KB3N1",
|
||||
"Y2Nlc3MYBiABKAgSFQoNZXJyb3JfbWVzc2FnZRgHIAEoCRItCgl0aW1lc3Rh",
|
||||
"bXAYCCABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wImQKF0RlYnVn",
|
||||
"U25hcHNob3RSZXF1ZXN0RHRvEhwKFGluc3RhbmNlX3VuaXF1ZV9uYW1lGAEg",
|
||||
"ASgJEhYKDmNvcnJlbGF0aW9uX2lkGAIgASgJEhMKC2FsYXJtc19vbmx5GAMg",
|
||||
"ASgIIlQKHFN1YnNjcmliZURlYnVnVmlld1JlcXVlc3REdG8SHAoUaW5zdGFu",
|
||||
"Y2VfdW5pcXVlX25hbWUYASABKAkSFgoOY29ycmVsYXRpb25faWQYAiABKAki",
|
||||
"VgoeVW5zdWJzY3JpYmVEZWJ1Z1ZpZXdSZXF1ZXN0RHRvEhwKFGluc3RhbmNl",
|
||||
"X3VuaXF1ZV9uYW1lGAEgASgJEhYKDmNvcnJlbGF0aW9uX2lkGAIgASgJIhwK",
|
||||
"GlVuc3Vic2NyaWJlRGVidWdWaWV3QWNrRHRvItQBChZBbGFybUNvbmRpdGlv",
|
||||
"blN0YXRlRHRvEg4KBmFjdGl2ZRgBIAEoCBIUCgxhY2tub3dsZWRnZWQYAiAB",
|
||||
"KAgSLQoJY29uZmlybWVkGAMgASgLMhouZ29vZ2xlLnByb3RvYnVmLkJvb2xW",
|
||||
"YWx1ZRI/CgZzaGVsdmUYBCABKA4yLy5zY2FkYWJyaWRnZS5zaXRlY29tbWFu",
|
||||
"ZC52MS5BbGFybVNoZWx2ZVN0YXRlRHRvEhIKCnN1cHByZXNzZWQYBSABKAgS",
|
||||
"EAoIc2V2ZXJpdHkYBiABKAUi3QEKFkRlYnVnQXR0cmlidXRlVmFsdWVEdG8S",
|
||||
"HAoUaW5zdGFuY2VfdW5pcXVlX25hbWUYASABKAkSFgoOYXR0cmlidXRlX3Bh",
|
||||
"dGgYAiABKAkSFgoOYXR0cmlidXRlX25hbWUYAyABKAkSNQoFdmFsdWUYBCAB",
|
||||
"KAsyJi5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5Mb29zZVZhbHVlEg8K",
|
||||
"B3F1YWxpdHkYBSABKAkSLQoJdGltZXN0YW1wGAYgASgLMhouZ29vZ2xlLnBy",
|
||||
"b3RvYnVmLlRpbWVzdGFtcCKvBQoSRGVidWdBbGFybVN0YXRlRHRvEhwKFGlu",
|
||||
"c3RhbmNlX3VuaXF1ZV9uYW1lGAEgASgJEhIKCmFsYXJtX25hbWUYAiABKAkS",
|
||||
"OAoFc3RhdGUYAyABKA4yKS5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5B",
|
||||
"bGFybVN0YXRlRHRvEhAKCHByaW9yaXR5GAQgASgFEi0KCXRpbWVzdGFtcBgF",
|
||||
"IAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXASOAoFbGV2ZWwYBiAB",
|
||||
"KA4yKS5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5BbGFybUxldmVsRHRv",
|
||||
"Eg8KB21lc3NhZ2UYByABKAkSNgoEa2luZBgIIAEoDjIoLnNjYWRhYnJpZGdl",
|
||||
"LnNpdGVjb21tYW5kLnYxLkFsYXJtS2luZER0bxJFCgljb25kaXRpb24YCSAB",
|
||||
"KAsyMi5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5BbGFybUNvbmRpdGlv",
|
||||
"blN0YXRlRHRvEhgKEHNvdXJjZV9yZWZlcmVuY2UYCiABKAkSFwoPYWxhcm1f",
|
||||
"dHlwZV9uYW1lGAsgASgJEhAKCGNhdGVnb3J5GAwgASgJEhUKDW9wZXJhdG9y",
|
||||
"X3VzZXIYDSABKAkSGAoQb3BlcmF0b3JfY29tbWVudBgOIAEoCRI3ChNvcmln",
|
||||
"aW5hbF9yYWlzZV90aW1lGA8gASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVz",
|
||||
"dGFtcBIVCg1jdXJyZW50X3ZhbHVlGBAgASgJEhMKC2xpbWl0X3ZhbHVlGBEg",
|
||||
"ASgJEiQKHG5hdGl2ZV9zb3VyY2VfY2Fub25pY2FsX25hbWUYEiABKAkSIQoZ",
|
||||
"aXNfY29uZmlndXJlZF9wbGFjZWhvbGRlchgTIAEoCCKcAgoURGVidWdWaWV3",
|
||||
"U25hcHNob3REdG8SHAoUaW5zdGFuY2VfdW5pcXVlX25hbWUYASABKAkSTAoQ",
|
||||
"YXR0cmlidXRlX3ZhbHVlcxgCIAMoCzIyLnNjYWRhYnJpZGdlLnNpdGVjb21t",
|
||||
"YW5kLnYxLkRlYnVnQXR0cmlidXRlVmFsdWVEdG8SRAoMYWxhcm1fc3RhdGVz",
|
||||
"GAMgAygLMi4uc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuRGVidWdBbGFy",
|
||||
"bVN0YXRlRHRvEjYKEnNuYXBzaG90X3RpbWVzdGFtcBgEIAEoCzIaLmdvb2ds",
|
||||
"ZS5wcm90b2J1Zi5UaW1lc3RhbXASGgoSaW5zdGFuY2Vfbm90X2ZvdW5kGAUg",
|
||||
"ASgIIvACCgxRdWVyeVJlcXVlc3QSTgoPZXZlbnRfbG9nX3F1ZXJ5GAEgASgL",
|
||||
"MjMuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuRXZlbnRMb2dRdWVyeVJl",
|
||||
"cXVlc3REdG9IABJNCg5kZWJ1Z19zbmFwc2hvdBgCIAEoCzIzLnNjYWRhYnJp",
|
||||
"ZGdlLnNpdGVjb21tYW5kLnYxLkRlYnVnU25hcHNob3RSZXF1ZXN0RHRvSAAS",
|
||||
"WAoUc3Vic2NyaWJlX2RlYnVnX3ZpZXcYAyABKAsyOC5zY2FkYWJyaWRnZS5z",
|
||||
"aXRlY29tbWFuZC52MS5TdWJzY3JpYmVEZWJ1Z1ZpZXdSZXF1ZXN0RHRvSAAS",
|
||||
"XAoWdW5zdWJzY3JpYmVfZGVidWdfdmlldxgEIAEoCzI6LnNjYWRhYnJpZGdl",
|
||||
"LnNpdGVjb21tYW5kLnYxLlVuc3Vic2NyaWJlRGVidWdWaWV3UmVxdWVzdER0",
|
||||
"b0gAQgkKB2NvbW1hbmQikQIKClF1ZXJ5UmVwbHkSTwoPZXZlbnRfbG9nX3F1",
|
||||
"ZXJ5GAEgASgLMjQuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuRXZlbnRM",
|
||||
"b2dRdWVyeVJlc3BvbnNlRHRvSAASTwoTZGVidWdfdmlld19zbmFwc2hvdBgC",
|
||||
"IAEoCzIwLnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLkRlYnVnVmlld1Nu",
|
||||
"YXBzaG90RHRvSAASWAoWdW5zdWJzY3JpYmVfZGVidWdfdmlldxgDIAEoCzI2",
|
||||
"LnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLlVuc3Vic2NyaWJlRGVidWdW",
|
||||
"aWV3QWNrRHRvSABCBwoFcmVwbHkingEKHFBhcmtlZE1lc3NhZ2VRdWVyeVJl",
|
||||
"cXVlc3REdG8SFgoOY29ycmVsYXRpb25faWQYASABKAkSDwoHc2l0ZV9pZBgC",
|
||||
"IAEoCRITCgtwYWdlX251bWJlchgDIAEoBRIRCglwYWdlX3NpemUYBCABKAUS",
|
||||
"LQoJdGltZXN0YW1wGAUgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFt",
|
||||
"cCLyAgoVUGFya2VkTWVzc2FnZUVudHJ5RHRvEhIKCm1lc3NhZ2VfaWQYASAB",
|
||||
"KAkSFQoNdGFyZ2V0X3N5c3RlbRgCIAEoCRITCgttZXRob2RfbmFtZRgDIAEo",
|
||||
"CRIVCg1lcnJvcl9tZXNzYWdlGAQgASgJEhUKDWF0dGVtcHRfY291bnQYBSAB",
|
||||
"KAUSNgoSb3JpZ2luYWxfdGltZXN0YW1wGAYgASgLMhouZ29vZ2xlLnByb3Rv",
|
||||
"YnVmLlRpbWVzdGFtcBI6ChZsYXN0X2F0dGVtcHRfdGltZXN0YW1wGAcgASgL",
|
||||
"MhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcBIUCgxtYXhfYXR0ZW1wdHMY",
|
||||
"CCABKAUSSAoIY2F0ZWdvcnkYCSABKA4yNi5zY2FkYWJyaWRnZS5zaXRlY29t",
|
||||
"bWFuZC52MS5TdG9yZUFuZEZvcndhcmRDYXRlZ29yeUR0bxIXCg9vcmlnaW5f",
|
||||
"aW5zdGFuY2UYCiABKAkioQIKHVBhcmtlZE1lc3NhZ2VRdWVyeVJlc3BvbnNl",
|
||||
"RHRvEhYKDmNvcnJlbGF0aW9uX2lkGAEgASgJEg8KB3NpdGVfaWQYAiABKAkS",
|
||||
"QwoIbWVzc2FnZXMYAyADKAsyMS5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52",
|
||||
"MS5QYXJrZWRNZXNzYWdlRW50cnlEdG8SEwoLdG90YWxfY291bnQYBCABKAUS",
|
||||
"EwoLcGFnZV9udW1iZXIYBSABKAUSEQoJcGFnZV9zaXplGAYgASgFEg8KB3N1",
|
||||
"Y2Nlc3MYByABKAgSFQoNZXJyb3JfbWVzc2FnZRgIIAEoCRItCgl0aW1lc3Rh",
|
||||
"bXAYCSABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wIooBChxQYXJr",
|
||||
"ZWRNZXNzYWdlUmV0cnlSZXF1ZXN0RHRvEhYKDmNvcnJlbGF0aW9uX2lkGAEg",
|
||||
"ASgJEg8KB3NpdGVfaWQYAiABKAkSEgoKbWVzc2FnZV9pZBgDIAEoCRItCgl0",
|
||||
"aW1lc3RhbXAYBCABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wIl8K",
|
||||
"HVBhcmtlZE1lc3NhZ2VSZXRyeVJlc3BvbnNlRHRvEhYKDmNvcnJlbGF0aW9u",
|
||||
"X2lkGAEgASgJEg8KB3N1Y2Nlc3MYAiABKAgSFQoNZXJyb3JfbWVzc2FnZRgD",
|
||||
"IAEoCSKMAQoeUGFya2VkTWVzc2FnZURpc2NhcmRSZXF1ZXN0RHRvEhYKDmNv",
|
||||
"cnJlbGF0aW9uX2lkGAEgASgJEg8KB3NpdGVfaWQYAiABKAkSEgoKbWVzc2Fn",
|
||||
"ZV9pZBgDIAEoCRItCgl0aW1lc3RhbXAYBCABKAsyGi5nb29nbGUucHJvdG9i",
|
||||
"dWYuVGltZXN0YW1wImEKH1BhcmtlZE1lc3NhZ2VEaXNjYXJkUmVzcG9uc2VE",
|
||||
"dG8SFgoOY29ycmVsYXRpb25faWQYASABKAkSDwoHc3VjY2VzcxgCIAEoCBIV",
|
||||
"Cg1lcnJvcl9tZXNzYWdlGAMgASgJIk8KF1JldHJ5UGFya2VkT3BlcmF0aW9u",
|
||||
"RHRvEhYKDmNvcnJlbGF0aW9uX2lkGAEgASgJEhwKFHRyYWNrZWRfb3BlcmF0",
|
||||
"aW9uX2lkGAIgASgJIlEKGURpc2NhcmRQYXJrZWRPcGVyYXRpb25EdG8SFgoO",
|
||||
"Y29ycmVsYXRpb25faWQYASABKAkSHAoUdHJhY2tlZF9vcGVyYXRpb25faWQY",
|
||||
"AiABKAkiXQobUGFya2VkT3BlcmF0aW9uQWN0aW9uQWNrRHRvEhYKDmNvcnJl",
|
||||
"bGF0aW9uX2lkGAEgASgJEg8KB2FwcGxpZWQYAiABKAgSFQoNZXJyb3JfbWVz",
|
||||
"c2FnZRgDIAEoCSLeAwoNUGFya2VkUmVxdWVzdBJYChRwYXJrZWRfbWVzc2Fn",
|
||||
"ZV9xdWVyeRgBIAEoCzI4LnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLlBh",
|
||||
"cmtlZE1lc3NhZ2VRdWVyeVJlcXVlc3REdG9IABJYChRwYXJrZWRfbWVzc2Fn",
|
||||
"ZV9yZXRyeRgCIAEoCzI4LnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLlBh",
|
||||
"cmtlZE1lc3NhZ2VSZXRyeVJlcXVlc3REdG9IABJcChZwYXJrZWRfbWVzc2Fn",
|
||||
"ZV9kaXNjYXJkGAMgASgLMjouc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEu",
|
||||
"UGFya2VkTWVzc2FnZURpc2NhcmRSZXF1ZXN0RHRvSAASVQoWcmV0cnlfcGFy",
|
||||
"a2VkX29wZXJhdGlvbhgEIAEoCzIzLnNjYWRhYnJpZGdlLnNpdGVjb21tYW5k",
|
||||
"LnYxLlJldHJ5UGFya2VkT3BlcmF0aW9uRHRvSAASWQoYZGlzY2FyZF9wYXJr",
|
||||
"ZWRfb3BlcmF0aW9uGAUgASgLMjUuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQu",
|
||||
"djEuRGlzY2FyZFBhcmtlZE9wZXJhdGlvbkR0b0gAQgkKB2NvbW1hbmQihwMK",
|
||||
"C1BhcmtlZFJlcGx5ElkKFHBhcmtlZF9tZXNzYWdlX3F1ZXJ5GAEgASgLMjku",
|
||||
"c2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUGFya2VkTWVzc2FnZVF1ZXJ5",
|
||||
"UmVzcG9uc2VEdG9IABJZChRwYXJrZWRfbWVzc2FnZV9yZXRyeRgCIAEoCzI5",
|
||||
"LnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLlBhcmtlZE1lc3NhZ2VSZXRy",
|
||||
"eVJlc3BvbnNlRHRvSAASXQoWcGFya2VkX21lc3NhZ2VfZGlzY2FyZBgDIAEo",
|
||||
"CzI7LnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLlBhcmtlZE1lc3NhZ2VE",
|
||||
"aXNjYXJkUmVzcG9uc2VEdG9IABJaChdwYXJrZWRfb3BlcmF0aW9uX2FjdGlv",
|
||||
"bhgEIAEoCzI3LnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLlBhcmtlZE9w",
|
||||
"ZXJhdGlvbkFjdGlvbkFja0R0b0gAQgcKBXJlcGx5IvwBChVSb3V0ZVRvQ2Fs",
|
||||
"bFJlcXVlc3REdG8SFgoOY29ycmVsYXRpb25faWQYASABKAkSHAoUaW5zdGFu",
|
||||
"Y2VfdW5pcXVlX25hbWUYAiABKAkSEwoLc2NyaXB0X25hbWUYAyABKAkSPQoK",
|
||||
"cGFyYW1ldGVycxgEIAEoCzIpLnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYx",
|
||||
"Lkxvb3NlVmFsdWVNYXASLQoJdGltZXN0YW1wGAUgASgLMhouZ29vZ2xlLnBy",
|
||||
"b3RvYnVmLlRpbWVzdGFtcBIbChNwYXJlbnRfZXhlY3V0aW9uX2lkGAYgASgJ",
|
||||
"Eg0KBXRyYWNlGAcgASgIIpICChZSb3V0ZVRvQ2FsbFJlc3BvbnNlRHRvEhYK",
|
||||
"DmNvcnJlbGF0aW9uX2lkGAEgASgJEg8KB3N1Y2Nlc3MYAiABKAgSPAoMcmV0",
|
||||
"dXJuX3ZhbHVlGAMgASgLMiYuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEu",
|
||||
"TG9vc2VWYWx1ZRIVCg1lcnJvcl9tZXNzYWdlGAQgASgJEi0KCXRpbWVzdGFt",
|
||||
"cBgFIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXASGgoSY2hpbGRf",
|
||||
"ZXhlY3V0aW9uX2lkGAYgASgJEi8KDHRyYWNlX2V2ZW50cxgHIAMoCzIZLnNp",
|
||||
"dGVzdHJlYW0uQXVkaXRFdmVudER0byK7AQoeUm91dGVUb0dldEF0dHJpYnV0",
|
||||
"ZXNSZXF1ZXN0RHRvEhYKDmNvcnJlbGF0aW9uX2lkGAEgASgJEhwKFGluc3Rh",
|
||||
"bmNlX3VuaXF1ZV9uYW1lGAIgASgJEhcKD2F0dHJpYnV0ZV9uYW1lcxgDIAMo",
|
||||
"CRItCgl0aW1lc3RhbXAYBCABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0",
|
||||
"YW1wEhsKE3BhcmVudF9leGVjdXRpb25faWQYBSABKAkiywEKH1JvdXRlVG9H",
|
||||
"ZXRBdHRyaWJ1dGVzUmVzcG9uc2VEdG8SFgoOY29ycmVsYXRpb25faWQYASAB",
|
||||
"KAkSOQoGdmFsdWVzGAIgASgLMikuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQu",
|
||||
"djEuTG9vc2VWYWx1ZU1hcBIPCgdzdWNjZXNzGAMgASgIEhUKDWVycm9yX21l",
|
||||
"c3NhZ2UYBCABKAkSLQoJdGltZXN0YW1wGAUgASgLMhouZ29vZ2xlLnByb3Rv",
|
||||
"YnVmLlRpbWVzdGFtcCLUAgoeUm91dGVUb1NldEF0dHJpYnV0ZXNSZXF1ZXN0",
|
||||
"RHRvEhYKDmNvcnJlbGF0aW9uX2lkGAEgASgJEhwKFGluc3RhbmNlX3VuaXF1",
|
||||
"ZV9uYW1lGAIgASgJEmkKEGF0dHJpYnV0ZV92YWx1ZXMYAyADKAsyTy5zY2Fk",
|
||||
"YWJyaWRnZS5zaXRlY29tbWFuZC52MS5Sb3V0ZVRvU2V0QXR0cmlidXRlc1Jl",
|
||||
"cXVlc3REdG8uQXR0cmlidXRlVmFsdWVzRW50cnkSLQoJdGltZXN0YW1wGAQg",
|
||||
"ASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcBIbChNwYXJlbnRfZXhl",
|
||||
"Y3V0aW9uX2lkGAUgASgJEg0KBXRyYWNlGAYgASgIGjYKFEF0dHJpYnV0ZVZh",
|
||||
"bHVlc0VudHJ5EgsKA2tleRgBIAEoCRINCgV2YWx1ZRgCIAEoCToCOAEikAEK",
|
||||
"H1JvdXRlVG9TZXRBdHRyaWJ1dGVzUmVzcG9uc2VEdG8SFgoOY29ycmVsYXRp",
|
||||
"b25faWQYASABKAkSDwoHc3VjY2VzcxgCIAEoCBIVCg1lcnJvcl9tZXNzYWdl",
|
||||
"GAMgASgJEi0KCXRpbWVzdGFtcBgEIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5U",
|
||||
"aW1lc3RhbXAiwwIKIVJvdXRlVG9XYWl0Rm9yQXR0cmlidXRlUmVxdWVzdER0",
|
||||
"bxIWCg5jb3JyZWxhdGlvbl9pZBgBIAEoCRIcChRpbnN0YW5jZV91bmlxdWVf",
|
||||
"bmFtZRgCIAEoCRIWCg5hdHRyaWJ1dGVfbmFtZRgDIAEoCRI6ChR0YXJnZXRf",
|
||||
"dmFsdWVfZW5jb2RlZBgEIAEoCzIcLmdvb2dsZS5wcm90b2J1Zi5TdHJpbmdW",
|
||||
"YWx1ZRIqCgd0aW1lb3V0GAUgASgLMhkuZ29vZ2xlLnByb3RvYnVmLkR1cmF0",
|
||||
"aW9uEi0KCXRpbWVzdGFtcBgGIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1l",
|
||||
"c3RhbXASGwoTcGFyZW50X2V4ZWN1dGlvbl9pZBgHIAEoCRIcChRyZXF1aXJl",
|
||||
"X2dvb2RfcXVhbGl0eRgIIAEoCCL/AQoiUm91dGVUb1dhaXRGb3JBdHRyaWJ1",
|
||||
"dGVSZXNwb25zZUR0bxIWCg5jb3JyZWxhdGlvbl9pZBgBIAEoCRIPCgdtYXRj",
|
||||
"aGVkGAIgASgIEjUKBXZhbHVlGAMgASgLMiYuc2NhZGFicmlkZ2Uuc2l0ZWNv",
|
||||
"bW1hbmQudjEuTG9vc2VWYWx1ZRIPCgdxdWFsaXR5GAQgASgJEhEKCXRpbWVk",
|
||||
"X291dBgFIAEoCBIPCgdzdWNjZXNzGAYgASgIEhUKDWVycm9yX21lc3NhZ2UY",
|
||||
"ByABKAkSLQoJdGltZXN0YW1wGAggASgLMhouZ29vZ2xlLnByb3RvYnVmLlRp",
|
||||
"bWVzdGFtcCKJAwoMUm91dGVSZXF1ZXN0EkoKDXJvdXRlX3RvX2NhbGwYASAB",
|
||||
"KAsyMS5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5Sb3V0ZVRvQ2FsbFJl",
|
||||
"cXVlc3REdG9IABJdChdyb3V0ZV90b19nZXRfYXR0cmlidXRlcxgCIAEoCzI6",
|
||||
"LnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLlJvdXRlVG9HZXRBdHRyaWJ1",
|
||||
"dGVzUmVxdWVzdER0b0gAEl0KF3JvdXRlX3RvX3NldF9hdHRyaWJ1dGVzGAMg",
|
||||
"ASgLMjouc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUm91dGVUb1NldEF0",
|
||||
"dHJpYnV0ZXNSZXF1ZXN0RHRvSAASZAobcm91dGVfdG9fd2FpdF9mb3JfYXR0",
|
||||
"cmlidXRlGAQgASgLMj0uc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUm91",
|
||||
"dGVUb1dhaXRGb3JBdHRyaWJ1dGVSZXF1ZXN0RHRvSABCCQoHY29tbWFuZCKJ",
|
||||
"AwoKUm91dGVSZXBseRJLCg1yb3V0ZV90b19jYWxsGAEgASgLMjIuc2NhZGFi",
|
||||
"cmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUm91dGVUb0NhbGxSZXNwb25zZUR0b0gA",
|
||||
"El4KF3JvdXRlX3RvX2dldF9hdHRyaWJ1dGVzGAIgASgLMjsuc2NhZGFicmlk",
|
||||
"Z2Uuc2l0ZWNvbW1hbmQudjEuUm91dGVUb0dldEF0dHJpYnV0ZXNSZXNwb25z",
|
||||
"ZUR0b0gAEl4KF3JvdXRlX3RvX3NldF9hdHRyaWJ1dGVzGAMgASgLMjsuc2Nh",
|
||||
"ZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUm91dGVUb1NldEF0dHJpYnV0ZXNS",
|
||||
"ZXNwb25zZUR0b0gAEmUKG3JvdXRlX3RvX3dhaXRfZm9yX2F0dHJpYnV0ZRgE",
|
||||
"IAEoCzI+LnNjYWRhYnJpZGdlLnNpdGVjb21tYW5kLnYxLlJvdXRlVG9XYWl0",
|
||||
"Rm9yQXR0cmlidXRlUmVzcG9uc2VEdG9IAEIHCgVyZXBseSJBChZUcmlnZ2Vy",
|
||||
"U2l0ZUZhaWxvdmVyRHRvEhYKDmNvcnJlbGF0aW9uX2lkGAEgASgJEg8KB3Np",
|
||||
"dGVfaWQYAiABKAkibQoSU2l0ZUZhaWxvdmVyQWNrRHRvEhYKDmNvcnJlbGF0",
|
||||
"aW9uX2lkGAEgASgJEhAKCGFjY2VwdGVkGAIgASgIEhYKDnRhcmdldF9hZGRy",
|
||||
"ZXNzGAMgASgJEhUKDWVycm9yX21lc3NhZ2UYBCABKAkqywEKE0RlcGxveW1l",
|
||||
"bnRTdGF0dXNEdG8SJQohREVQTE9ZTUVOVF9TVEFUVVNfRFRPX1VOU1BFQ0lG",
|
||||
"SUVEEAASIQodREVQTE9ZTUVOVF9TVEFUVVNfRFRPX1BFTkRJTkcQARIlCiFE",
|
||||
"RVBMT1lNRU5UX1NUQVRVU19EVE9fSU5fUFJPR1JFU1MQAhIhCh1ERVBMT1lN",
|
||||
"RU5UX1NUQVRVU19EVE9fU1VDQ0VTUxADEiAKHERFUExPWU1FTlRfU1RBVFVT",
|
||||
"X0RUT19GQUlMRUQQBCrEAQoSQnJvd3NlTm9kZUNsYXNzRHRvEiUKIUJST1dT",
|
||||
"RV9OT0RFX0NMQVNTX0RUT19VTlNQRUNJRklFRBAAEiAKHEJST1dTRV9OT0RF",
|
||||
"X0NMQVNTX0RUT19PQkpFQ1QQARIiCh5CUk9XU0VfTk9ERV9DTEFTU19EVE9f",
|
||||
"VkFSSUFCTEUQAhIgChxCUk9XU0VfTk9ERV9DTEFTU19EVE9fTUVUSE9EEAMS",
|
||||
"HwobQlJPV1NFX05PREVfQ0xBU1NfRFRPX09USEVSEAQqoQIKFEJyb3dzZUZh",
|
||||
"aWx1cmVLaW5kRHRvEicKI0JST1dTRV9GQUlMVVJFX0tJTkRfRFRPX1VOU1BF",
|
||||
"Q0lGSUVEEAASMAosQlJPV1NFX0ZBSUxVUkVfS0lORF9EVE9fQ09OTkVDVElP",
|
||||
"Tl9OT1RfRk9VTkQQARI0CjBCUk9XU0VfRkFJTFVSRV9LSU5EX0RUT19DT05O",
|
||||
"RUNUSU9OX05PVF9DT05ORUNURUQQAhIpCiVCUk9XU0VfRkFJTFVSRV9LSU5E",
|
||||
"X0RUT19OT1RfQlJPV1NBQkxFEAMSIwofQlJPV1NFX0ZBSUxVUkVfS0lORF9E",
|
||||
"VE9fVElNRU9VVBAEEigKJEJST1dTRV9GQUlMVVJFX0tJTkRfRFRPX1NFUlZF",
|
||||
"Ul9FUlJPUhAFKqoCChtSZWFkVGFnVmFsdWVzRmFpbHVyZUtpbmREdG8SMAos",
|
||||
"UkVBRF9UQUdfVkFMVUVTX0ZBSUxVUkVfS0lORF9EVE9fVU5TUEVDSUZJRUQQ",
|
||||
"ABI5CjVSRUFEX1RBR19WQUxVRVNfRkFJTFVSRV9LSU5EX0RUT19DT05ORUNU",
|
||||
"SU9OX05PVF9GT1VORBABEj0KOVJFQURfVEFHX1ZBTFVFU19GQUlMVVJFX0tJ",
|
||||
"TkRfRFRPX0NPTk5FQ1RJT05fTk9UX0NPTk5FQ1RFRBACEiwKKFJFQURfVEFH",
|
||||
"X1ZBTFVFU19GQUlMVVJFX0tJTkRfRFRPX1RJTUVPVVQQAxIxCi1SRUFEX1RB",
|
||||
"R19WQUxVRVNfRkFJTFVSRV9LSU5EX0RUT19TRVJWRVJfRVJST1IQBCqTAgoU",
|
||||
"VmVyaWZ5RmFpbHVyZUtpbmREdG8SJwojVkVSSUZZX0ZBSUxVUkVfS0lORF9E",
|
||||
"VE9fVU5TUEVDSUZJRUQQABInCiNWRVJJRllfRkFJTFVSRV9LSU5EX0RUT19V",
|
||||
"TlJFQUNIQUJMRRABEicKI1ZFUklGWV9GQUlMVVJFX0tJTkRfRFRPX0FVVEhf",
|
||||
"RkFJTEVEEAISMQotVkVSSUZZX0ZBSUxVUkVfS0lORF9EVE9fVU5UUlVTVEVE",
|
||||
"X0NFUlRJRklDQVRFEAMSIwofVkVSSUZZX0ZBSUxVUkVfS0lORF9EVE9fVElN",
|
||||
"RU9VVBAEEigKJFZFUklGWV9GQUlMVVJFX0tJTkRfRFRPX1NFUlZFUl9FUlJP",
|
||||
"UhAFKuUBChpTdG9yZUFuZEZvcndhcmRDYXRlZ29yeUR0bxIuCipTVE9SRV9B",
|
||||
"TkRfRk9SV0FSRF9DQVRFR09SWV9EVE9fVU5TUEVDSUZJRUQQABIyCi5TVE9S",
|
||||
"RV9BTkRfRk9SV0FSRF9DQVRFR09SWV9EVE9fRVhURVJOQUxfU1lTVEVNEAES",
|
||||
"LworU1RPUkVfQU5EX0ZPUldBUkRfQ0FURUdPUllfRFRPX05PVElGSUNBVElP",
|
||||
"ThACEjIKLlNUT1JFX0FORF9GT1JXQVJEX0NBVEVHT1JZX0RUT19DQUNIRURf",
|
||||
"REJfV1JJVEUQAypoCg1BbGFybVN0YXRlRHRvEh8KG0FMQVJNX1NUQVRFX0RU",
|
||||
"T19VTlNQRUNJRklFRBAAEhoKFkFMQVJNX1NUQVRFX0RUT19BQ1RJVkUQARIa",
|
||||
"ChZBTEFSTV9TVEFURV9EVE9fTk9STUFMEAIquQEKDUFsYXJtTGV2ZWxEdG8S",
|
||||
"HwobQUxBUk1fTEVWRUxfRFRPX1VOU1BFQ0lGSUVEEAASGAoUQUxBUk1fTEVW",
|
||||
"RUxfRFRPX05PTkUQARIXChNBTEFSTV9MRVZFTF9EVE9fTE9XEAISGwoXQUxB",
|
||||
"Uk1fTEVWRUxfRFRPX0xPV19MT1cQAxIYChRBTEFSTV9MRVZFTF9EVE9fSElH",
|
||||
"SBAEEh0KGUFMQVJNX0xFVkVMX0RUT19ISUdIX0hJR0gQBSqSAQoMQWxhcm1L",
|
||||
"aW5kRHRvEh4KGkFMQVJNX0tJTkRfRFRPX1VOU1BFQ0lGSUVEEAASGwoXQUxB",
|
||||
"Uk1fS0lORF9EVE9fQ09NUFVURUQQARIgChxBTEFSTV9LSU5EX0RUT19OQVRJ",
|
||||
"VkVfT1BDX1VBEAISIwofQUxBUk1fS0lORF9EVE9fTkFUSVZFX01YX0FDQ0VT",
|
||||
"UxADKugBChNBbGFybVNoZWx2ZVN0YXRlRHRvEiYKIkFMQVJNX1NIRUxWRV9T",
|
||||
"VEFURV9EVE9fVU5TUEVDSUZJRUQQABIkCiBBTEFSTV9TSEVMVkVfU1RBVEVf",
|
||||
"RFRPX1VOU0hFTFZFRBABEisKJ0FMQVJNX1NIRUxWRV9TVEFURV9EVE9fT05F",
|
||||
"X1NIT1RfU0hFTFZFRBACEigKJEFMQVJNX1NIRUxWRV9TVEFURV9EVE9fVElN",
|
||||
"RURfU0hFTFZFRBADEiwKKEFMQVJNX1NIRUxWRV9TVEFURV9EVE9fUEVSTUFO",
|
||||
"RU5UX1NIRUxWRUQQBDKEBQoSU2l0ZUNvbW1hbmRTZXJ2aWNlEmwKEEV4ZWN1",
|
||||
"dGVMaWZlY3ljbGUSLC5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5MaWZl",
|
||||
"Y3ljbGVSZXF1ZXN0Giouc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuTGlm",
|
||||
"ZWN5Y2xlUmVwbHkSYAoMRXhlY3V0ZU9wY1VhEiguc2NhZGFicmlkZ2Uuc2l0",
|
||||
"ZWNvbW1hbmQudjEuT3BjVWFSZXF1ZXN0GiYuc2NhZGFicmlkZ2Uuc2l0ZWNv",
|
||||
"bW1hbmQudjEuT3BjVWFSZXBseRJgCgxFeGVjdXRlUXVlcnkSKC5zY2FkYWJy",
|
||||
"aWRnZS5zaXRlY29tbWFuZC52MS5RdWVyeVJlcXVlc3QaJi5zY2FkYWJyaWRn",
|
||||
"ZS5zaXRlY29tbWFuZC52MS5RdWVyeVJlcGx5EmMKDUV4ZWN1dGVQYXJrZWQS",
|
||||
"KS5zY2FkYWJyaWRnZS5zaXRlY29tbWFuZC52MS5QYXJrZWRSZXF1ZXN0Gicu",
|
||||
"c2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUGFya2VkUmVwbHkSYAoMRXhl",
|
||||
"Y3V0ZVJvdXRlEiguc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUm91dGVS",
|
||||
"ZXF1ZXN0GiYuc2NhZGFicmlkZ2Uuc2l0ZWNvbW1hbmQudjEuUm91dGVSZXBs",
|
||||
"eRJ1Cg9UcmlnZ2VyRmFpbG92ZXISMi5zY2FkYWJyaWRnZS5zaXRlY29tbWFu",
|
||||
"ZC52MS5UcmlnZ2VyU2l0ZUZhaWxvdmVyRHRvGi4uc2NhZGFicmlkZ2Uuc2l0",
|
||||
"ZWNvbW1hbmQudjEuU2l0ZUZhaWxvdmVyQWNrRHRvQiuqAihaQi5NT00uV1cu",
|
||||
"U2NhZGFCcmlkZ2UuQ29tbXVuaWNhdGlvbi5HcnBjYgZwcm90bzM="));
|
||||
descriptor = pbr::FileDescriptor.FromGeneratedCode(descriptorData,
|
||||
new pbr::FileDescriptor[] { global::Google.Protobuf.WellKnownTypes.TimestampReflection.Descriptor, global::Google.Protobuf.WellKnownTypes.DurationReflection.Descriptor, global::Google.Protobuf.WellKnownTypes.WrappersReflection.Descriptor, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor, },
|
||||
new pbr::GeneratedClrTypeInfo(new[] {typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.DeploymentStatusDto), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.BrowseNodeClassDto), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.BrowseFailureKindDto), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ReadTagValuesFailureKindDto), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.VerifyFailureKindDto), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.StoreAndForwardCategoryDto), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AlarmStateDto), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AlarmLevelDto), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AlarmKindDto), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AlarmShelveStateDto), }, null, new pbr::GeneratedClrTypeInfo[] {
|
||||
@@ -543,7 +543,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.DeploymentStatusResponseDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.DeploymentStatusResponseDto.Parser, new[]{ "DeploymentId", "InstanceUniqueName", "Status", "ErrorMessage", "Timestamp" }, null, null, null, null),
|
||||
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.InstanceLifecycleResponseDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.InstanceLifecycleResponseDto.Parser, new[]{ "CommandId", "InstanceUniqueName", "Success", "ErrorMessage", "Timestamp" }, null, null, null, null),
|
||||
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.DeploymentStateQueryResponseDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.DeploymentStateQueryResponseDto.Parser, new[]{ "CorrelationId", "InstanceUniqueName", "IsDeployed", "AppliedDeploymentId", "AppliedRevisionHash", "Timestamp" }, null, null, null, null),
|
||||
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ArtifactDeploymentResponseDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ArtifactDeploymentResponseDto.Parser, new[]{ "DeploymentId", "SiteId", "Success", "ErrorMessage", "Timestamp", "SecretsApplied" }, null, null, null, null),
|
||||
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ArtifactDeploymentResponseDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ArtifactDeploymentResponseDto.Parser, new[]{ "DeploymentId", "SiteId", "Success", "ErrorMessage", "Timestamp", "SecretsApplied", "SecretsRemoved" }, null, null, null, null),
|
||||
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleRequest), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleRequest.Parser, new[]{ "RefreshDeployment", "EnableInstance", "DisableInstance", "DeleteInstance", "DeploymentStateQuery", "DeployArtifacts" }, new[]{ "Command" }, null, null, null),
|
||||
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleReply), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.LifecycleReply.Parser, new[]{ "DeploymentStatus", "InstanceLifecycle", "DeploymentStateQuery", "ArtifactDeployment" }, new[]{ "Reply" }, null, null, null),
|
||||
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.BrowseNodeCommandDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.BrowseNodeCommandDto.Parser, new[]{ "ConnectionName", "ParentNodeId", "ContinuationToken", "SiteIdentifier" }, null, null, null, null),
|
||||
@@ -9751,6 +9751,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
errorMessage_ = other.errorMessage_;
|
||||
timestamp_ = other.timestamp_ != null ? other.timestamp_.Clone() : null;
|
||||
SecretsApplied = other.SecretsApplied;
|
||||
secretsRemoved_ = other.secretsRemoved_.Clone();
|
||||
_unknownFields = pb::UnknownFieldSet.Clone(other._unknownFields);
|
||||
}
|
||||
|
||||
@@ -9840,6 +9841,20 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
}
|
||||
|
||||
|
||||
/// <summary>Field number for the "secrets_removed" field.</summary>
|
||||
public const int SecretsRemovedFieldNumber = 7;
|
||||
private static readonly pb::FieldCodec<string> _repeated_secretsRemoved_codec
|
||||
= pb::FieldCodec.ForString(58);
|
||||
private readonly pbc::RepeatedField<string> secretsRemoved_ = new pbc::RepeatedField<string>();
|
||||
/// <summary>
|
||||
/// names only; empty => null
|
||||
/// </summary>
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
public pbc::RepeatedField<string> SecretsRemoved {
|
||||
get { return secretsRemoved_; }
|
||||
}
|
||||
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
public override bool Equals(object other) {
|
||||
@@ -9861,6 +9876,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
if (ErrorMessage != other.ErrorMessage) return false;
|
||||
if (!object.Equals(Timestamp, other.Timestamp)) return false;
|
||||
if (SecretsApplied != other.SecretsApplied) return false;
|
||||
if(!secretsRemoved_.Equals(other.secretsRemoved_)) return false;
|
||||
return Equals(_unknownFields, other._unknownFields);
|
||||
}
|
||||
|
||||
@@ -9874,6 +9890,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
if (ErrorMessage.Length != 0) hash ^= ErrorMessage.GetHashCode();
|
||||
if (timestamp_ != null) hash ^= Timestamp.GetHashCode();
|
||||
if (secretsApplied_ != null) hash ^= SecretsApplied.GetHashCode();
|
||||
hash ^= secretsRemoved_.GetHashCode();
|
||||
if (_unknownFields != null) {
|
||||
hash ^= _unknownFields.GetHashCode();
|
||||
}
|
||||
@@ -9915,6 +9932,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
if (secretsApplied_ != null) {
|
||||
_single_secretsApplied_codec.WriteTagAndValue(output, SecretsApplied);
|
||||
}
|
||||
secretsRemoved_.WriteTo(output, _repeated_secretsRemoved_codec);
|
||||
if (_unknownFields != null) {
|
||||
_unknownFields.WriteTo(output);
|
||||
}
|
||||
@@ -9948,6 +9966,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
if (secretsApplied_ != null) {
|
||||
_single_secretsApplied_codec.WriteTagAndValue(ref output, SecretsApplied);
|
||||
}
|
||||
secretsRemoved_.WriteTo(ref output, _repeated_secretsRemoved_codec);
|
||||
if (_unknownFields != null) {
|
||||
_unknownFields.WriteTo(ref output);
|
||||
}
|
||||
@@ -9976,6 +9995,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
if (secretsApplied_ != null) {
|
||||
size += _single_secretsApplied_codec.CalculateSizeWithTag(SecretsApplied);
|
||||
}
|
||||
size += secretsRemoved_.CalculateSize(_repeated_secretsRemoved_codec);
|
||||
if (_unknownFields != null) {
|
||||
size += _unknownFields.CalculateSize();
|
||||
}
|
||||
@@ -10011,6 +10031,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
SecretsApplied = other.SecretsApplied;
|
||||
}
|
||||
}
|
||||
secretsRemoved_.Add(other.secretsRemoved_);
|
||||
_unknownFields = pb::UnknownFieldSet.MergeFrom(_unknownFields, other._unknownFields);
|
||||
}
|
||||
|
||||
@@ -10060,6 +10081,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
}
|
||||
break;
|
||||
}
|
||||
case 58: {
|
||||
secretsRemoved_.AddEntriesFrom(input, _repeated_secretsRemoved_codec);
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
#endif
|
||||
@@ -10109,6 +10134,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
}
|
||||
break;
|
||||
}
|
||||
case 58: {
|
||||
secretsRemoved_.AddEntriesFrom(ref input, _repeated_secretsRemoved_codec);
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -111,10 +111,11 @@ public class ArtifactDeploymentService
|
||||
/// single-site retries).
|
||||
/// </param>
|
||||
/// <returns>
|
||||
/// The deployment artifacts command for the site, and any secret references its data
|
||||
/// connections make that could not be resolved at central (absent or tombstoned). A
|
||||
/// non-empty <c>Missing</c> list means the command must NOT be sent to the site —
|
||||
/// see <see cref="DeployCoreAsync"/>/<see cref="RetryForSiteAsync"/>.
|
||||
/// The deployment artifacts command for the site, any secret references its data
|
||||
/// connections make that could not be resolved at central (absent or tombstoned), and the
|
||||
/// central secret store's read error if resolution could not complete. A non-empty
|
||||
/// <c>Missing</c> list or a non-null <c>SecretStoreError</c> means the command must NOT be
|
||||
/// sent to the site — see <see cref="DeployCoreAsync"/>/<see cref="RetryForSiteAsync"/>.
|
||||
/// </returns>
|
||||
/// <remarks>
|
||||
/// This convenience overload runs the global artifact queries
|
||||
@@ -123,7 +124,7 @@ public class ArtifactDeploymentService
|
||||
/// per-site loop and calls the prefetched-globals overload to avoid the N+1
|
||||
/// re-query of every system-wide artifact set per site.
|
||||
/// </remarks>
|
||||
public async Task<(DeployArtifactsCommand Command, IReadOnlyList<SecretReference> Missing)> BuildDeployArtifactsCommandAsync(
|
||||
public async Task<(DeployArtifactsCommand Command, IReadOnlyList<SecretReference> Missing, string? SecretStoreError)> BuildDeployArtifactsCommandAsync(
|
||||
int siteId,
|
||||
CancellationToken cancellationToken = default,
|
||||
string? deploymentId = null)
|
||||
@@ -157,7 +158,7 @@ public class ArtifactDeploymentService
|
||||
/// distinguishable marker into the hash even for a command the caller ultimately refuses
|
||||
/// to send (see the returned <c>Missing</c> tuple element).
|
||||
/// </remarks>
|
||||
private async Task<(DeployArtifactsCommand Command, IReadOnlyList<SecretReference> Missing)> BuildDeployArtifactsCommandAsync(
|
||||
private async Task<(DeployArtifactsCommand Command, IReadOnlyList<SecretReference> Missing, string? SecretStoreError)> BuildDeployArtifactsCommandAsync(
|
||||
int siteId,
|
||||
GlobalArtifactSnapshot globals,
|
||||
CancellationToken cancellationToken,
|
||||
@@ -193,7 +194,7 @@ public class ArtifactDeploymentService
|
||||
Secrets: collected.Secrets,
|
||||
MissingSecrets: collected.Missing.Count > 0 ? collected.Missing.Select(m => m.Name).ToList() : null);
|
||||
|
||||
return (command, collected.Missing);
|
||||
return (command, collected.Missing, collected.StoreError);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -221,19 +222,24 @@ public class ArtifactDeploymentService
|
||||
var shipped = cmd.Secrets?.Count ?? 0;
|
||||
if (r.Success && shipped > 0 && r.SecretsApplied is null)
|
||||
return new(siteIdentifier, siteName, false,
|
||||
"Site build predates secret deployment — upgrade the site before deploying secrets", null);
|
||||
"Site build predates secret deployment — upgrade the site before deploying secrets", null,
|
||||
r.SecretsRemoved);
|
||||
if (r.Success && shipped > 0 && r.SecretsApplied != shipped)
|
||||
return new(siteIdentifier, siteName, false,
|
||||
$"Site applied {r.SecretsApplied} of {shipped} secrets", r.SecretsApplied);
|
||||
return new(siteIdentifier, siteName, r.Success, r.ErrorMessage, shipped > 0 ? r.SecretsApplied : null);
|
||||
$"Site applied {r.SecretsApplied} of {shipped} secrets", r.SecretsApplied, r.SecretsRemoved);
|
||||
return new(siteIdentifier, siteName, r.Success, r.ErrorMessage, shipped > 0 ? r.SecretsApplied : null,
|
||||
r.SecretsRemoved);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Best-effort <c>DeploySiteSecrets</c> audit row (task 7): one row per site whose command
|
||||
/// carried, or would have carried, at least one secret — a resolved
|
||||
/// <see cref="DeployArtifactsCommand.Secrets"/> entry, an unresolved reference in
|
||||
/// <paramref name="missing"/> (the fail-closed case, where nothing was sent), or both. A
|
||||
/// site with no secret references at all gets no row.
|
||||
/// <paramref name="missing"/> (the fail-closed case, where nothing was sent), a central
|
||||
/// secret-store read failure (<paramref name="secretStoreError"/>, also fail-closed), or a
|
||||
/// site-reported removal of previously-deployed secrets (<paramref name="removed"/>, the
|
||||
/// site's <see cref="ArtifactDeploymentResponse.SecretsRemoved"/>). A site with none of
|
||||
/// these gets no row.
|
||||
///
|
||||
/// Per the audit-is-best-effort rule (an audit failure never changes the deploy outcome)
|
||||
/// this is fire-and-forget-shaped only in intent, not mechanism — like every other audit
|
||||
@@ -251,13 +257,22 @@ public class ArtifactDeploymentService
|
||||
string user,
|
||||
DeployArtifactsCommand command,
|
||||
IReadOnlyList<SecretReference> missing,
|
||||
string? secretStoreError,
|
||||
IReadOnlyList<string>? removed,
|
||||
string siteIdentifier,
|
||||
string siteName,
|
||||
bool success,
|
||||
string? error,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var hasSecretRefs = (command.Secrets?.Count ?? 0) > 0 || missing.Count > 0;
|
||||
// A removal-only deploy (the site's last secret reference was dropped, so nothing
|
||||
// ships but the site retires previously-deployed rows) and a secret-store outage both
|
||||
// count as "secrets involved" — the former used to write no row at all (final review
|
||||
// M6 follow-up), leaving the removal visible only in the site's own logs.
|
||||
var hasSecretRefs = (command.Secrets?.Count ?? 0) > 0
|
||||
|| missing.Count > 0
|
||||
|| secretStoreError is not null
|
||||
|| removed is { Count: > 0 };
|
||||
if (!hasSecretRefs)
|
||||
return Task.CompletedTask;
|
||||
|
||||
@@ -270,6 +285,7 @@ public class ArtifactDeploymentService
|
||||
.Select(s => new { name = s.Name, revision = s.Revision })
|
||||
.ToList(),
|
||||
missing = missing.Select(m => m.Name).ToList(),
|
||||
removed = (removed ?? []).ToList(),
|
||||
success,
|
||||
error,
|
||||
},
|
||||
@@ -427,7 +443,7 @@ public class ArtifactDeploymentService
|
||||
// Each entry also carries the site's unresolved secret references (design §3.3):
|
||||
// a non-empty Missing list means this site's command must never reach
|
||||
// _communicationService — see the fan-out loop below.
|
||||
var siteCommands = new Dictionary<int, (DeployArtifactsCommand Command, IReadOnlyList<SecretReference> Missing)>();
|
||||
var siteCommands = new Dictionary<int, (DeployArtifactsCommand Command, IReadOnlyList<SecretReference> Missing, string? SecretStoreError)>();
|
||||
foreach (var site in sites)
|
||||
{
|
||||
siteCommands[site.Id] = await BuildDeployArtifactsCommandAsync(
|
||||
@@ -457,7 +473,19 @@ public class ArtifactDeploymentService
|
||||
using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
|
||||
cts.CancelAfter(_options.ArtifactDeploymentTimeoutPerSite);
|
||||
|
||||
var (command, missing) = siteCommands[site.Id];
|
||||
var (command, missing, secretStoreError) = siteCommands[site.Id];
|
||||
|
||||
// A central secret-store read failure (final review M5) fails only the sites
|
||||
// that reference a secret — resolution is incomplete, so shipping would either
|
||||
// drop a secret the site still needs or retire one it still uses. Sites with no
|
||||
// secret references never touch the store and deploy normally.
|
||||
if (secretStoreError is not null)
|
||||
{
|
||||
_logger.LogError(
|
||||
"Cannot deploy artifacts to site {SiteId} ({SiteName}): {Error}",
|
||||
site.SiteIdentifier, site.Name, secretStoreError);
|
||||
return new SiteArtifactResult(site.SiteIdentifier, site.Name, false, secretStoreError);
|
||||
}
|
||||
|
||||
// Fail this site closed BEFORE anything is sent (design §3.3): a referenced
|
||||
// secret that is absent or tombstoned at central must not let the rest of the
|
||||
@@ -528,11 +556,11 @@ public class ArtifactDeploymentService
|
||||
// rule — BEFORE the best-effort stamp below.
|
||||
foreach (var site in sites)
|
||||
{
|
||||
var (command, missing) = siteCommands[site.Id];
|
||||
var (command, missing, secretStoreError) = siteCommands[site.Id];
|
||||
var result = perSiteResults[site.SiteIdentifier];
|
||||
await LogSecretsAuditAsync(
|
||||
user, command, missing, site.SiteIdentifier, site.Name,
|
||||
result.Success, result.ErrorMessage, cancellationToken);
|
||||
user, command, missing, secretStoreError, result.SecretsRemoved,
|
||||
site.SiteIdentifier, site.Name, result.Success, result.ErrorMessage, cancellationToken);
|
||||
}
|
||||
|
||||
// Persist the system artifact deployment record.
|
||||
@@ -611,7 +639,18 @@ public class ArtifactDeploymentService
|
||||
using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
|
||||
cts.CancelAfter(_options.ArtifactDeploymentTimeoutPerSite);
|
||||
|
||||
var (command, missing) = await BuildDeployArtifactsCommandAsync(siteDbId, cts.Token);
|
||||
var (command, missing, secretStoreError) = await BuildDeployArtifactsCommandAsync(siteDbId, cts.Token);
|
||||
|
||||
// Same store-outage rule as the sweep path (final review M5).
|
||||
if (secretStoreError is not null)
|
||||
{
|
||||
_logger.LogError(
|
||||
"Cannot deploy artifacts to site {SiteId}: {Error}", siteIdentifier, secretStoreError);
|
||||
await LogSecretsAuditAsync(
|
||||
user, command, missing, secretStoreError, null, siteIdentifier, siteIdentifier,
|
||||
false, secretStoreError, cancellationToken);
|
||||
return Result<SiteArtifactResult>.Failure(secretStoreError);
|
||||
}
|
||||
|
||||
// Same fail-closed rule as the sweep path (design §3.3): a retry must not ship a
|
||||
// site an artifact set whose secret references still don't resolve.
|
||||
@@ -623,7 +662,7 @@ public class ArtifactDeploymentService
|
||||
// Task 7: the fail-closed case still gets its DeploySiteSecrets row — nothing
|
||||
// was sent, but the command carried (or would have carried) secrets.
|
||||
await LogSecretsAuditAsync(
|
||||
user, command, missing, siteIdentifier, siteIdentifier, false, missingMessage,
|
||||
user, command, missing, null, null, siteIdentifier, siteIdentifier, false, missingMessage,
|
||||
cancellationToken);
|
||||
return Result<SiteArtifactResult>.Failure(missingMessage);
|
||||
}
|
||||
@@ -648,7 +687,7 @@ public class ArtifactDeploymentService
|
||||
|
||||
// Ordering rule: audit before the best-effort stamp.
|
||||
await LogSecretsAuditAsync(
|
||||
user, command, missing, siteIdentifier, siteIdentifier,
|
||||
user, command, missing, null, result.SecretsRemoved, siteIdentifier, siteIdentifier,
|
||||
result.Success, result.ErrorMessage, cancellationToken);
|
||||
|
||||
// Same stamp-on-success rule as the sweep path: a retry that lands is a
|
||||
@@ -699,6 +738,12 @@ public class ArtifactDeploymentService
|
||||
/// would silently report unqueried sites as up to date. The pending-updates cache discards a
|
||||
/// faulted flight and the UI shows the aggregate as unavailable instead.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// A central secret-store read failure is the one per-site exception (final review M5): the
|
||||
/// affected site is left OUT of the map, which the evaluator reads as <i>Out of date</i> (a
|
||||
/// missing entry never matches a stamp), so a store outage degrades only the sites that
|
||||
/// reference secrets instead of failing the fleet-wide check.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public async Task<IReadOnlyDictionary<int, string>> ComputeCurrentArtifactHashesAsync(
|
||||
IReadOnlyList<Site> sites,
|
||||
@@ -717,8 +762,22 @@ public class ArtifactDeploymentService
|
||||
// MissingSecrets straight off the command (design §3.4), so an unresolved
|
||||
// reference still folds a distinguishable marker into the hash — the site reads
|
||||
// *Out of date* instead of this call failing the whole batch.
|
||||
var (command, _) = await BuildDeployArtifactsCommandAsync(
|
||||
var (command, _, secretStoreError) = await BuildDeployArtifactsCommandAsync(
|
||||
site.Id, globals, cancellationToken, deploymentId: "current");
|
||||
|
||||
// A central secret-store outage (final review M5) leaves this site's current hash
|
||||
// unknowable. Omitting it from the map makes the site read *Out of date* — the
|
||||
// conservative answer, never a false *Current* — while every other site, including
|
||||
// every site with no secret references, still evaluates normally instead of the
|
||||
// whole fleet check failing.
|
||||
if (secretStoreError is not null)
|
||||
{
|
||||
_logger.LogWarning(
|
||||
"Could not compute the current artifact hash for site {SiteId}: {Error}",
|
||||
site.SiteIdentifier, secretStoreError);
|
||||
continue;
|
||||
}
|
||||
|
||||
hashes[site.Id] = _hashService.ComputeHash(command);
|
||||
}
|
||||
|
||||
@@ -800,12 +859,19 @@ public class ArtifactDeploymentService
|
||||
/// doesn't know about the field) are indistinguishable, which is correct: neither one is a
|
||||
/// secrets-gating concern.
|
||||
/// </param>
|
||||
/// <param name="SecretsRemoved">
|
||||
/// The previously-deployed secret names the site retired on this deploy because no data
|
||||
/// connection references them any more (the site's
|
||||
/// <see cref="ArtifactDeploymentResponse.SecretsRemoved"/>); <c>null</c> when the site reported
|
||||
/// none or predates the field. Feeds the <c>DeploySiteSecrets</c> audit row's <c>removed</c> list.
|
||||
/// </param>
|
||||
public record SiteArtifactResult(
|
||||
string SiteId,
|
||||
string SiteName,
|
||||
bool Success,
|
||||
string? ErrorMessage,
|
||||
int? SecretsDeployed = null);
|
||||
int? SecretsDeployed = null,
|
||||
IReadOnlyList<string>? SecretsRemoved = null);
|
||||
|
||||
/// <summary>
|
||||
/// Summary of system-wide artifact deployment with per-site results.
|
||||
|
||||
@@ -6,12 +6,23 @@ namespace ZB.MOM.WW.ScadaBridge.DeploymentManager;
|
||||
|
||||
/// <summary>
|
||||
/// Result of <see cref="SiteSecretsCollector.CollectAsync"/>: the resolved ciphertext rows for a
|
||||
/// site's referenced secrets, and the references that could not be resolved (absent or
|
||||
/// tombstoned) at central.
|
||||
/// site's referenced secrets, the references that could not be resolved (absent or
|
||||
/// tombstoned) at central, and — when the central secret store itself could not be read — the
|
||||
/// error that stopped resolution.
|
||||
/// </summary>
|
||||
/// <param name="Secrets">The resolved rows.</param>
|
||||
/// <param name="Missing">References that are absent or tombstoned in the store.</param>
|
||||
/// <param name="StoreError">
|
||||
/// <see langword="null"/> when every reference was looked up. Otherwise the store read failed
|
||||
/// (final review M5: e.g. the shared SQL Server secret store is unreachable), resolution stopped
|
||||
/// at the first failure, and <see cref="Secrets"/>/<see cref="Missing"/> are INCOMPLETE — the
|
||||
/// caller must treat this site as undeployable and its current hash as unknown. The message is
|
||||
/// the exception type and message only; it never contains secret material.
|
||||
/// </param>
|
||||
public sealed record SiteSecretsCollection(
|
||||
IReadOnlyList<SecretArtifact> Secrets,
|
||||
IReadOnlyList<SecretReference> Missing);
|
||||
IReadOnlyList<SecretReference> Missing,
|
||||
string? StoreError = null);
|
||||
|
||||
/// <summary>
|
||||
/// Central half of design §3.1/§3.3: derives which secrets a site's data connections reference
|
||||
@@ -30,7 +41,11 @@ public sealed class SiteSecretsCollector(ISecretStore store)
|
||||
/// <param name="ct">A token to cancel the operation.</param>
|
||||
/// <returns>
|
||||
/// The resolved rows (mapped field-for-field from <see cref="StoredSecret"/>) and the
|
||||
/// references that were absent or tombstoned in the store.
|
||||
/// references that were absent or tombstoned in the store. A store read failure is
|
||||
/// returned as <see cref="SiteSecretsCollection.StoreError"/> rather than thrown, so one
|
||||
/// site's lookup failing never aborts a whole multi-site sweep (final review M5); only
|
||||
/// cancellation of <paramref name="ct"/> propagates. A site whose connections reference no
|
||||
/// secret never touches the store at all.
|
||||
/// </returns>
|
||||
public async Task<SiteSecretsCollection> CollectAsync(
|
||||
IEnumerable<DataConnectionArtifact> connections, CancellationToken ct)
|
||||
@@ -40,7 +55,19 @@ public sealed class SiteSecretsCollector(ISecretStore store)
|
||||
|
||||
foreach (var reference in SecretReferenceScanner.Scan(connections))
|
||||
{
|
||||
var row = await store.GetAsync(new SecretName(reference.Name), ct);
|
||||
StoredSecret? row;
|
||||
try
|
||||
{
|
||||
row = await store.GetAsync(new SecretName(reference.Name), ct);
|
||||
}
|
||||
catch (Exception ex) when (!ct.IsCancellationRequested)
|
||||
{
|
||||
return new SiteSecretsCollection(
|
||||
secrets, missing,
|
||||
$"Central secret store could not be read while resolving '{reference.Name}' " +
|
||||
$"({ex.GetType().Name}: {ex.Message})");
|
||||
}
|
||||
|
||||
if (row is null || row.IsDeleted)
|
||||
{
|
||||
missing.Add(reference);
|
||||
|
||||
@@ -198,6 +198,19 @@ public interface ISiteHealthCollector
|
||||
// Default no-op so test fakes do not need to be updated.
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Replace the latest per-node deployed-secrets mirror statuses (review finding M1) used
|
||||
/// by the <c>SecretMirrorStatuses</c> field of the next <see cref="CollectReport"/> call.
|
||||
/// Refreshed periodically by the site-side <c>SecretMirrorStatusReporter</c> from the
|
||||
/// LocalDb-replicated <c>secret_mirror_status</c> table, so the ACTIVE node's report
|
||||
/// carries the STANDBY node's mirror failures too. Point-in-time: NOT reset on collect.
|
||||
/// </summary>
|
||||
/// <param name="statuses">Every node's row, or <see langword="null"/> for "no data".</param>
|
||||
void SetSecretMirrorStatuses(IReadOnlyList<SecretMirrorNodeStatus>? statuses)
|
||||
{
|
||||
// Default no-op so test fakes do not need to be updated.
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Replace the latest script-execution-scheduler gauges (queue depth, busy
|
||||
/// thread count, and age in seconds of the oldest in-flight script) used by
|
||||
|
||||
@@ -32,6 +32,10 @@ public class SiteHealthCollector : ISiteHealthCollector
|
||||
// -1 = reconciler SKIPPED (no IDeploymentConfigFetcher)
|
||||
// Null must stay distinct from -1: "we don't know" is not "self-heal is missing".
|
||||
private volatile int _startupReconcilerActive;
|
||||
// Review finding M1: every site node's deployed-secrets mirror status, refreshed by the
|
||||
// SiteRuntime SecretMirrorStatusReporter. Null = the reporter has not run (or its read
|
||||
// failed before it ever succeeded) — "no data", distinct from an empty list.
|
||||
private volatile IReadOnlyList<SecretMirrorNodeStatus>? _secretMirrorStatuses;
|
||||
private readonly ConcurrentDictionary<string, ConnectionHealth> _connectionStatuses = new();
|
||||
private readonly ConcurrentDictionary<string, TagResolutionStatus> _tagResolutionCounts = new();
|
||||
private readonly ConcurrentDictionary<string, string> _connectionEndpoints = new();
|
||||
@@ -130,6 +134,12 @@ public class SiteHealthCollector : ISiteHealthCollector
|
||||
_startupReconcilerActive = active ? 1 : -1;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public void SetSecretMirrorStatuses(IReadOnlyList<SecretMirrorNodeStatus>? statuses)
|
||||
{
|
||||
_secretMirrorStatuses = statuses?.ToList();
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public void UpdateConnectionHealth(string connectionName, ConnectionHealth health)
|
||||
{
|
||||
@@ -333,7 +343,9 @@ public class SiteHealthCollector : ISiteHealthCollector
|
||||
SiteStreamAlarmDropCount = Interlocked.Read(ref _siteStreamAlarmDropCount),
|
||||
// Tri-state: 0 stays null ("not reported"), which must never collapse into
|
||||
// false ("self-heal is missing"). Register row 73.
|
||||
StartupReconcilerActive = startupReconciler == 0 ? null : startupReconciler > 0
|
||||
StartupReconcilerActive = startupReconciler == 0 ? null : startupReconciler > 0,
|
||||
// M1: point-in-time, not drained; null stays null ("no data").
|
||||
SecretMirrorStatuses = _secretMirrorStatuses
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
@@ -48,7 +48,7 @@ namespace ZB.MOM.WW.ScadaBridge.Host;
|
||||
public static class SiteLocalDbSetup
|
||||
{
|
||||
/// <summary>
|
||||
/// The eleven tables that participate in replication — registered when this node has a peer,
|
||||
/// The twelve tables that participate in replication — registered when this node has a peer,
|
||||
/// and deregistered (stale triggers dropped) when it does not.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
@@ -69,6 +69,13 @@ public static class SiteLocalDbSetup
|
||||
/// <c>deployed_secrets</c> row, and replicating the ledger would make one node's progress
|
||||
/// masquerade as the other's.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <c>secret_mirror_status</c> is the twelfth entry (review finding M1). It is per-node
|
||||
/// DATA — one row per node, keyed by <c>node_name</c>, each written only by its own node —
|
||||
/// but it is replicated precisely so the active node can read the standby's row and put
|
||||
/// it on the site health report (only the active node sends one). Explicit primary key,
|
||||
/// no FK, no BLOB columns.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
internal static readonly string[] ReplicatedTables =
|
||||
[
|
||||
@@ -83,6 +90,7 @@ public static class SiteLocalDbSetup
|
||||
"data_connection_definitions",
|
||||
"native_alarm_state",
|
||||
"deployed_secrets",
|
||||
"secret_mirror_status",
|
||||
];
|
||||
|
||||
/// <summary>
|
||||
@@ -140,8 +148,9 @@ public static class SiteLocalDbSetup
|
||||
// ELEVENTH entry to ReplicatedTables on an already-replicating site streams a
|
||||
// snapshot of only that one table, where through 0.2.x the flag was a single
|
||||
// per-database bit and the same edit would have re-streamed all eleven in full, in
|
||||
// both directions. Nothing about the first boot changes: enabling replication on a
|
||||
// site that has been running without it seeds all eleven at once, the flagged set is
|
||||
// both directions (the same holds for secret_mirror_status, the twelfth, added by
|
||||
// review finding M1). Nothing about the first boot changes: enabling replication on a
|
||||
// site that has been running without it seeds all twelve at once, the flagged set is
|
||||
// therefore every registered table, and the library sends that as an ordinary full
|
||||
// snapshot. A hypothetical "register the whole list in one call" API would silently
|
||||
// give the coarse behaviour back.
|
||||
|
||||
@@ -3352,6 +3352,7 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
|
||||
// registered on this node fails the deploy rather than silently applying
|
||||
// nothing.
|
||||
int? secretsApplied = null;
|
||||
IReadOnlyList<string>? secretsRemoved = null;
|
||||
string? secretError = null;
|
||||
if (command.Secrets != null)
|
||||
{
|
||||
@@ -3371,7 +3372,20 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
|
||||
_logger.LogInformation(
|
||||
"Applying {Count} deployed secret(s), deploymentId={DeploymentId}",
|
||||
command.Secrets.Count, command.DeploymentId);
|
||||
await _storage.UpsertDeployedSecretsAsync(command.Secrets, command.DeploymentId);
|
||||
var (_, removedNames) =
|
||||
await _storage.UpsertDeployedSecretsAsync(command.Secrets, command.DeploymentId);
|
||||
// Reported to central for its DeploySiteSecrets audit row, so a deploy
|
||||
// that only retires secrets is still audited there. These are the
|
||||
// deployed_secrets rows this deploy marked removed (site-wide, the
|
||||
// authoritative "no longer deployed" set) — not whichever of them this
|
||||
// node's mirror happened to own and delete.
|
||||
if (removedNames.Count > 0)
|
||||
{
|
||||
secretsRemoved = removedNames;
|
||||
_logger.LogInformation(
|
||||
"Retired {Count} deployed secret(s) no longer referenced, deploymentId={DeploymentId}: {Names}",
|
||||
removedNames.Count, command.DeploymentId, string.Join(", ", removedNames));
|
||||
}
|
||||
var mirror = await _secretsMirror.ReconcileAsync(CancellationToken.None);
|
||||
|
||||
// The mirror reconciles the WHOLE site-wide deployed_secrets table
|
||||
@@ -3418,7 +3432,7 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
|
||||
|
||||
return new ArtifactDeploymentResponse(
|
||||
command.DeploymentId, "", secretError is null, secretError, DateTimeOffset.UtcNow,
|
||||
secretsApplied);
|
||||
secretsApplied, secretsRemoved);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
|
||||
@@ -3,8 +3,8 @@ using Microsoft.Data.Sqlite;
|
||||
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
|
||||
|
||||
/// <summary>
|
||||
/// DDL for the site's eleven configuration tables (nine original plus
|
||||
/// <c>deployed_secrets</c> and <c>mirrored_secrets</c>), extracted from
|
||||
/// DDL for the site's twelve configuration tables (nine original plus
|
||||
/// <c>deployed_secrets</c>, <c>mirrored_secrets</c> and <c>secret_mirror_status</c>), extracted from
|
||||
/// <see cref="SiteStorageService"/> so it can be applied by whoever owns the database
|
||||
/// file.
|
||||
/// <para>
|
||||
@@ -23,7 +23,7 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
|
||||
public static class SiteStorageSchema
|
||||
{
|
||||
/// <summary>
|
||||
/// Creates the eleven site configuration tables when absent, and additively upgrades
|
||||
/// Creates the twelve site configuration tables when absent, and additively upgrades
|
||||
/// tables created by an older build. Idempotent — safe to run on every startup.
|
||||
/// </summary>
|
||||
/// <param name="connection">An open connection to the database that should hold the tables.</param>
|
||||
@@ -136,6 +136,20 @@ public static class SiteStorageSchema
|
||||
revision INTEGER NOT NULL,
|
||||
mirrored_at TEXT NOT NULL
|
||||
);
|
||||
|
||||
-- Per-node deployed-secrets mirror status (review finding M1): one row per site
|
||||
-- node, keyed by node name, holding that node's CURRENT mirror failures as a JSON
|
||||
-- array of {SecretName, Message} (names and messages only — never values or
|
||||
-- ciphertext). Unlike mirrored_secrets this IS replicated
|
||||
-- (SiteLocalDbSetup.ReplicatedTables): each node writes only its own row, and the
|
||||
-- replication is exactly what lets the ACTIVE node's health report carry the
|
||||
-- STANDBY node's failures to central. Written only when a node's failure set
|
||||
-- changes (plus once per start-up), never per mirror tick. No foreign keys.
|
||||
CREATE TABLE IF NOT EXISTS secret_mirror_status (
|
||||
node_name TEXT PRIMARY KEY,
|
||||
failures_json TEXT NOT NULL,
|
||||
checked_at TEXT NOT NULL
|
||||
);
|
||||
";
|
||||
command.ExecuteNonQuery();
|
||||
}
|
||||
|
||||
@@ -3,6 +3,7 @@ using Microsoft.Data.Sqlite;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using ZB.MOM.WW.LocalDb;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
|
||||
|
||||
@@ -924,6 +925,87 @@ public class SiteStorageService
|
||||
await command.ExecuteNonQueryAsync();
|
||||
}
|
||||
|
||||
// ── Secret mirror status (review finding M1 — LocalDb-replicated, one row per node) ──
|
||||
|
||||
/// <summary>
|
||||
/// Records <paramref name="nodeName"/>'s current deployed-secrets mirror failures in the
|
||||
/// replicated <c>secret_mirror_status</c> table (UPSERT on the node name). An empty
|
||||
/// <paramref name="failures"/> list records a healthy mirror. Every call is a replicated
|
||||
/// write, so callers write only when the failure set changes.
|
||||
/// </summary>
|
||||
/// <param name="nodeName">This node's name — the row key.</param>
|
||||
/// <param name="failures">The node's current failures (names and messages only, never values).</param>
|
||||
/// <param name="checkedAtUtc">When the recorded failure set was observed.</param>
|
||||
/// <returns>A task that completes when the row has been stored.</returns>
|
||||
public async Task SetSecretMirrorStatusAsync(
|
||||
string nodeName, IReadOnlyList<SecretMirrorFailure> failures, DateTimeOffset checkedAtUtc)
|
||||
{
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(nodeName);
|
||||
ArgumentNullException.ThrowIfNull(failures);
|
||||
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var command = connection.CreateCommand();
|
||||
command.CommandText = @"
|
||||
INSERT INTO secret_mirror_status (node_name, failures_json, checked_at)
|
||||
VALUES (@nodeName, @failures, @checkedAt)
|
||||
ON CONFLICT(node_name) DO UPDATE SET
|
||||
failures_json = excluded.failures_json,
|
||||
checked_at = excluded.checked_at";
|
||||
|
||||
command.Parameters.AddWithValue("@nodeName", nodeName);
|
||||
command.Parameters.AddWithValue("@failures", JsonSerializer.Serialize(failures));
|
||||
command.Parameters.AddWithValue("@checkedAt", checkedAtUtc.ToUniversalTime().ToString("O"));
|
||||
|
||||
await command.ExecuteNonQueryAsync();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Returns every node's row from <c>secret_mirror_status</c>, ordered by node name. A row
|
||||
/// whose <c>failures_json</c> cannot be parsed is returned with a single synthetic failure
|
||||
/// saying so, rather than failing the whole read or silently reading as healthy.
|
||||
/// </summary>
|
||||
/// <returns>A task that resolves to every node's recorded mirror status.</returns>
|
||||
public async Task<IReadOnlyList<SecretMirrorNodeStatus>> GetSecretMirrorStatusesAsync()
|
||||
{
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var command = connection.CreateCommand();
|
||||
command.CommandText =
|
||||
"SELECT node_name, failures_json, checked_at FROM secret_mirror_status ORDER BY node_name";
|
||||
|
||||
var results = new List<SecretMirrorNodeStatus>();
|
||||
await using var reader = await command.ExecuteReaderAsync();
|
||||
while (await reader.ReadAsync())
|
||||
{
|
||||
var nodeName = reader.GetString(0);
|
||||
IReadOnlyList<SecretMirrorFailure> failures;
|
||||
try
|
||||
{
|
||||
failures = JsonSerializer.Deserialize<List<SecretMirrorFailure>>(reader.GetString(1))
|
||||
?? [];
|
||||
}
|
||||
catch (Exception ex) when (ex is JsonException or NotSupportedException)
|
||||
{
|
||||
failures =
|
||||
[
|
||||
new SecretMirrorFailure(
|
||||
"(unreadable)",
|
||||
$"Mirror status row for node '{nodeName}' could not be parsed ({ex.GetType().Name})"),
|
||||
];
|
||||
}
|
||||
|
||||
var checkedAt = DateTimeOffset.TryParse(
|
||||
reader.GetString(2), null, System.Globalization.DateTimeStyles.RoundtripKind, out var parsed)
|
||||
? parsed
|
||||
: DateTimeOffset.MinValue;
|
||||
|
||||
results.Add(new SecretMirrorNodeStatus(nodeName, checkedAt, failures));
|
||||
}
|
||||
|
||||
return results;
|
||||
}
|
||||
|
||||
// ── Shared Script CRUD ──
|
||||
|
||||
/// <summary>
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
|
||||
@@ -153,6 +155,18 @@ public sealed record MirrorResult(
|
||||
/// so a LATER failure — even one with an identical fingerprint — logs at Error again. This state
|
||||
/// is in-memory only (per node process), which matches the ledger's own node-local scope.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Status row for central (review finding M1).</b> A failure on the STANDBY node used to be
|
||||
/// visible only in that node's own log — site health reports are sent by the active node only.
|
||||
/// After each pass this mirror records its node's current failure set (secret name + message,
|
||||
/// never a value) in the LocalDb-replicated <c>secret_mirror_status</c> table, keyed by node
|
||||
/// name; <c>SecretMirrorStatusReporter</c> on the active node reads every node's row onto the
|
||||
/// health report. Because every write replicates via CDC, the row is written only when the
|
||||
/// failure set (compared as sorted name+message pairs) differs from what this process last
|
||||
/// wrote — plus once on the first pass after start-up, so a restart refreshes a stale row. The
|
||||
/// write is best-effort: a failure logs a Warning (repeats at Debug) and is retried on the next
|
||||
/// pass, and never affects the pass's own <see cref="MirrorResult"/>.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class DeployedSecretsMirror : BackgroundService
|
||||
{
|
||||
@@ -165,6 +179,17 @@ public sealed class DeployedSecretsMirror : BackgroundService
|
||||
private readonly IOptions<SiteRuntimeOptions> _options;
|
||||
private readonly ILogger<DeployedSecretsMirror> _logger;
|
||||
private readonly TimeProvider _time;
|
||||
private readonly string _nodeName;
|
||||
|
||||
/// <summary>
|
||||
/// The failure set (sorted name+message pairs) this process last wrote successfully to
|
||||
/// <c>secret_mirror_status</c>, or <see langword="null"/> before the first successful write
|
||||
/// — which is what forces the once-per-start-up write (review finding M1).
|
||||
/// </summary>
|
||||
private IReadOnlyList<SecretMirrorFailure>? _lastWrittenStatus;
|
||||
|
||||
/// <summary>Whether a failed status write has already been logged at Warning (de-spam).</summary>
|
||||
private bool _statusWriteFailureLogged;
|
||||
|
||||
private readonly SemaphoreSlim _gate = new(1, 1);
|
||||
private bool _disposed;
|
||||
@@ -199,6 +224,11 @@ public sealed class DeployedSecretsMirror : BackgroundService
|
||||
/// <param name="options">Supplies <see cref="SiteRuntimeOptions.DeployedSecretsMirrorInterval"/>.</param>
|
||||
/// <param name="logger">Logger instance. Never logs secret values — names, revisions, and KEK ids only.</param>
|
||||
/// <param name="time">Clock driving the reconcile loop's <see cref="PeriodicTimer"/> and the tombstone-recovery stamp.</param>
|
||||
/// <param name="nodeIdentity">
|
||||
/// Supplies this node's name (<c>node-a</c>/<c>node-b</c>), the key of its
|
||||
/// <c>secret_mirror_status</c> row. Optional: when absent or unconfigured, the machine name
|
||||
/// is used instead.
|
||||
/// </param>
|
||||
public DeployedSecretsMirror(
|
||||
SiteStorageService storage,
|
||||
ISecretStore store,
|
||||
@@ -206,7 +236,8 @@ public sealed class DeployedSecretsMirror : BackgroundService
|
||||
ISecretCacheInvalidator? invalidator,
|
||||
IOptions<SiteRuntimeOptions> options,
|
||||
ILogger<DeployedSecretsMirror> logger,
|
||||
TimeProvider time)
|
||||
TimeProvider time,
|
||||
INodeIdentityProvider? nodeIdentity = null)
|
||||
{
|
||||
_storage = storage ?? throw new ArgumentNullException(nameof(storage));
|
||||
_store = store ?? throw new ArgumentNullException(nameof(store));
|
||||
@@ -215,6 +246,7 @@ public sealed class DeployedSecretsMirror : BackgroundService
|
||||
_options = options ?? throw new ArgumentNullException(nameof(options));
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
_time = time ?? throw new ArgumentNullException(nameof(time));
|
||||
_nodeName = nodeIdentity?.NodeName ?? Environment.MachineName;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -314,6 +346,8 @@ public sealed class DeployedSecretsMirror : BackgroundService
|
||||
rows.Count(r => !r.IsRemoved) + unreadable.Count(u => !u.IsRemoved), failed.Count);
|
||||
}
|
||||
|
||||
await RecordStatusIfChangedAsync(failedNames, failed).ConfigureAwait(false);
|
||||
|
||||
return result;
|
||||
}
|
||||
finally
|
||||
@@ -322,6 +356,42 @@ public sealed class DeployedSecretsMirror : BackgroundService
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Writes this node's <c>secret_mirror_status</c> row when the pass's failure set differs
|
||||
/// from the last one this process wrote (or nothing has been written yet since start-up).
|
||||
/// Best-effort: never throws, so the pass result is unaffected by a status-write fault.
|
||||
/// See the class remarks (review finding M1).
|
||||
/// </summary>
|
||||
private async Task RecordStatusIfChangedAsync(IReadOnlyList<string> failedNames, IReadOnlyList<string> failed)
|
||||
{
|
||||
var current = failedNames
|
||||
.Select((name, i) => new SecretMirrorFailure(name, failed[i]))
|
||||
.OrderBy(f => f.SecretName, StringComparer.Ordinal)
|
||||
.ThenBy(f => f.Message, StringComparer.Ordinal)
|
||||
.ToList();
|
||||
|
||||
if (_lastWrittenStatus is not null && _lastWrittenStatus.SequenceEqual(current))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
await _storage.SetSecretMirrorStatusAsync(_nodeName, current, _time.GetUtcNow()).ConfigureAwait(false);
|
||||
_lastWrittenStatus = current;
|
||||
_statusWriteFailureLogged = false;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.Log(
|
||||
_statusWriteFailureLogged ? LogLevel.Debug : LogLevel.Warning,
|
||||
ex,
|
||||
"Failed to record the deployed secrets mirror status for node {NodeName}; retrying next pass",
|
||||
_nodeName);
|
||||
_statusWriteFailureLogged = true;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Handles a tombstoned <c>deployed_secrets</c> row: deletes it from this node's store, but
|
||||
/// only if this node had previously ledgered it as mirrored — an out-of-band, same-named
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using ZB.MOM.WW.ScadaBridge.HealthMonitoring;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Secrets;
|
||||
|
||||
/// <summary>
|
||||
/// Review finding M1: site-side hosted service that periodically reads EVERY node's row from
|
||||
/// the LocalDb-replicated <c>secret_mirror_status</c> table and pushes the list into
|
||||
/// <see cref="ISiteHealthCollector.SetSecretMirrorStatuses"/>, so the next health report
|
||||
/// carries each site node's deployed-secrets mirror failures — the standby's included.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// Runs on every site node (the read is a cheap local SELECT), but only the active node's
|
||||
/// report is ever sent (<c>HealthReportSender</c>), which is exactly why the status lives in a
|
||||
/// REPLICATED table: the standby writes its row, CDC carries it to the active node, and this
|
||||
/// reporter on the active node forwards it. A site pair WITHOUT LocalDb replication reports
|
||||
/// only the active node's row — the same prerequisite the deployed secrets themselves have.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Lives in SiteRuntime (which already references HealthMonitoring and owns
|
||||
/// <see cref="SiteStorageService"/>), following <c>SiteStreamAlarmDropReporter</c>: immediate
|
||||
/// first probe, fixed cadence, exceptions logged and swallowed. A failed read leaves the
|
||||
/// previous snapshot on the collector rather than clearing it.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class SecretMirrorStatusReporter : BackgroundService
|
||||
{
|
||||
/// <summary>Default poll cadence (30 s), matching the site health report interval.</summary>
|
||||
internal static readonly TimeSpan DefaultPollInterval = TimeSpan.FromSeconds(30);
|
||||
|
||||
private readonly ISiteHealthCollector _collector;
|
||||
private readonly SiteStorageService _storage;
|
||||
private readonly ILogger<SecretMirrorStatusReporter> _logger;
|
||||
private readonly TimeSpan _pollInterval;
|
||||
|
||||
/// <summary>Initializes a new instance of <see cref="SecretMirrorStatusReporter"/>.</summary>
|
||||
/// <param name="collector">The site health collector receiving the statuses.</param>
|
||||
/// <param name="storage">Site storage exposing <c>secret_mirror_status</c>.</param>
|
||||
/// <param name="logger">Logger instance.</param>
|
||||
/// <param name="pollInterval">Poll interval override; defaults to <see cref="DefaultPollInterval"/> (30 s).</param>
|
||||
public SecretMirrorStatusReporter(
|
||||
ISiteHealthCollector collector,
|
||||
SiteStorageService storage,
|
||||
ILogger<SecretMirrorStatusReporter> logger,
|
||||
TimeSpan? pollInterval = null)
|
||||
{
|
||||
_collector = collector ?? throw new ArgumentNullException(nameof(collector));
|
||||
_storage = storage ?? throw new ArgumentNullException(nameof(storage));
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
_pollInterval = pollInterval ?? DefaultPollInterval;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
|
||||
{
|
||||
await ProbeAsync().ConfigureAwait(false);
|
||||
|
||||
while (!stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
try
|
||||
{
|
||||
await Task.Delay(_pollInterval, stoppingToken).ConfigureAwait(false);
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
break;
|
||||
}
|
||||
|
||||
await ProbeAsync().ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Reads every node's mirror status and pushes it onto the collector; never throws.
|
||||
/// Public so a test can drive one deterministic probe.
|
||||
/// </summary>
|
||||
/// <returns>A task that completes when the probe has finished.</returns>
|
||||
public async Task ProbeAsync()
|
||||
{
|
||||
try
|
||||
{
|
||||
var statuses = await _storage.GetSecretMirrorStatusesAsync().ConfigureAwait(false);
|
||||
_collector.SetSecretMirrorStatuses(statuses);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
// Catch-all is deliberate: health reporting must never take the node down, and
|
||||
// the loop must survive so the next tick retries.
|
||||
_logger.LogWarning(ex, "SecretMirrorStatusReporter probe failed; next tick will retry.");
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -140,6 +140,13 @@ public static class ServiceCollectionExtensions
|
||||
services.AddSingleton<DeployedSecretsMirror>();
|
||||
services.AddHostedService(sp => sp.GetRequiredService<DeployedSecretsMirror>());
|
||||
|
||||
// Review finding M1: forward every node's replicated secret_mirror_status row onto the
|
||||
// site health report, so the ACTIVE node's report carries the STANDBY's mirror failures.
|
||||
services.AddHostedService(sp => new SecretMirrorStatusReporter(
|
||||
sp.GetRequiredService<HealthMonitoring.ISiteHealthCollector>(),
|
||||
sp.GetRequiredService<SiteStorageService>(),
|
||||
sp.GetRequiredService<ILogger<SecretMirrorStatusReporter>>()));
|
||||
|
||||
return services;
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,66 @@
|
||||
using System.CommandLine;
|
||||
using System.Net;
|
||||
using System.Net.Sockets;
|
||||
using System.Text;
|
||||
using ZB.MOM.WW.ScadaBridge.CLI.Commands;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.CLI.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Both artifact-deploy commands (<c>site deploy-artifacts</c> and <c>deploy artifacts</c>) exit
|
||||
/// 1 when any site in the summary failed. A per-site failure is a successful HTTP call, so they
|
||||
/// used to exit 0 and a script could not tell a failed site from a clean deploy.
|
||||
/// </summary>
|
||||
public class DeployArtifactsExitCodeTests
|
||||
{
|
||||
private const string AllSucceeded =
|
||||
"{\"deploymentId\":\"d1\",\"siteResults\":[{\"siteId\":\"site-a\",\"siteName\":\"A\",\"success\":true}]," +
|
||||
"\"successCount\":1,\"failureCount\":0}";
|
||||
|
||||
private const string OneFailed =
|
||||
"{\"deploymentId\":\"d1\",\"siteResults\":[{\"siteId\":\"site-a\",\"siteName\":\"A\",\"success\":true}," +
|
||||
"{\"siteId\":\"site-b\",\"siteName\":\"B\",\"success\":false,\"errorMessage\":\"Secret 'k' ... missing\"}]," +
|
||||
"\"successCount\":1,\"failureCount\":1}";
|
||||
|
||||
[Theory]
|
||||
[InlineData("site", AllSucceeded, 0)]
|
||||
[InlineData("site", OneFailed, 1)]
|
||||
[InlineData("site", "{\"unexpected\":true}", 1)]
|
||||
[InlineData("deploy", AllSucceeded, 0)]
|
||||
[InlineData("deploy", OneFailed, 1)]
|
||||
[InlineData("deploy", "{\"unexpected\":true}", 1)]
|
||||
public async Task ArtifactDeployExitCodeFollowsFailureCount(string group, string response, int expected)
|
||||
{
|
||||
var listener = new TcpListener(IPAddress.Loopback, 0);
|
||||
listener.Start();
|
||||
try
|
||||
{
|
||||
var port = ((IPEndPoint)listener.LocalEndpoint).Port;
|
||||
var server = Task.Run(async () =>
|
||||
{
|
||||
using var client = await listener.AcceptTcpClientAsync();
|
||||
await using var stream = client.GetStream();
|
||||
using var reader = new StreamReader(stream, Encoding.ASCII, leaveOpen: true);
|
||||
while (await reader.ReadLineAsync() is { Length: > 0 }) { }
|
||||
var body = Encoding.UTF8.GetBytes(response);
|
||||
var headers = Encoding.ASCII.GetBytes($"HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {body.Length}\r\nConnection: close\r\n\r\n");
|
||||
await stream.WriteAsync(headers);
|
||||
await stream.WriteAsync(body);
|
||||
});
|
||||
var root = new RootCommand();
|
||||
var url = new Option<string>("--url") { Recursive = true };
|
||||
var format = new Option<string>("--format") { Recursive = true };
|
||||
var user = new Option<string>("--username") { Recursive = true };
|
||||
var password = new Option<string>("--password") { Recursive = true };
|
||||
root.Add(url); root.Add(format); root.Add(user); root.Add(password);
|
||||
root.Add(SiteCommands.Build(url, format, user, password));
|
||||
root.Add(DeployCommands.Build(url, format, user, password));
|
||||
string[] verb = group == "site" ? ["site", "deploy-artifacts"] : ["deploy", "artifacts"];
|
||||
var exit = await root.Parse([.. verb, "--url", $"http://127.0.0.1:{port}",
|
||||
"--username", "test", "--password", "test", "--format", "json"]).InvokeAsync();
|
||||
await server.WaitAsync(TimeSpan.FromSeconds(5));
|
||||
Assert.Equal(expected, exit);
|
||||
}
|
||||
finally { listener.Stop(); }
|
||||
}
|
||||
}
|
||||
@@ -50,7 +50,11 @@ public class DeploymentDeadlineParityTests
|
||||
using var payload = JsonDocument.Parse(new string(request));
|
||||
Assert.Equal(expectedCommand, payload.RootElement.GetProperty("command").GetString());
|
||||
await Task.Delay(TimeSpan.FromSeconds(31), deadline.Token);
|
||||
var reply = Encoding.ASCII.GetBytes("HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: 2\r\nConnection: close\r\n\r\n{}");
|
||||
// A zero-failure summary: the artifact-deploy aliases derive their exit code from
|
||||
// failureCount, so a bare "{}" would read as an invalid summary (exit 1).
|
||||
const string body = "{\"successCount\":1,\"failureCount\":0}";
|
||||
var reply = Encoding.ASCII.GetBytes(
|
||||
$"HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {body.Length}\r\nConnection: close\r\n\r\n{body}");
|
||||
try { await stream.WriteAsync(reply, deadline.Token); }
|
||||
catch (IOException) { /* Old client closes at 30s; exit assertion diagnoses it. */ }
|
||||
}, deadline.Token);
|
||||
|
||||
@@ -559,6 +559,70 @@ public class HealthPageTests : BunitContext
|
||||
StartupReconcilerActive = active,
|
||||
};
|
||||
|
||||
// ----- Review finding M1: per-node deployed-secrets mirror failures ------------------
|
||||
//
|
||||
// The standby's failures ride the active node's report; the card must name the node,
|
||||
// the secret and the message. Null ("no data") and all-healthy nodes render nothing.
|
||||
|
||||
[Fact]
|
||||
public void Renders_SecretMirrorFailures_LabelledByNode()
|
||||
{
|
||||
SeedStates(new SiteHealthState
|
||||
{
|
||||
SiteId = "site-a",
|
||||
IsOnline = true,
|
||||
LastHeartbeatAt = DateTimeOffset.UtcNow,
|
||||
LatestReport = ReportWithReconciler("site-a", active: true) with
|
||||
{
|
||||
SecretMirrorStatuses =
|
||||
[
|
||||
new SecretMirrorNodeStatus("node-a", DateTimeOffset.UtcNow, []),
|
||||
new SecretMirrorNodeStatus(
|
||||
"node-b",
|
||||
DateTimeOffset.UtcNow,
|
||||
[new SecretMirrorFailure("OPC-PWD", "did not decrypt on this node")]),
|
||||
],
|
||||
},
|
||||
});
|
||||
|
||||
var cut = RenderHealthPage();
|
||||
|
||||
cut.WaitForAssertion(() =>
|
||||
{
|
||||
Assert.Contains("data-test=\"secret-mirror-badge\"", cut.Markup);
|
||||
var panel = cut.Find("[data-test=secret-mirror-failures]");
|
||||
Assert.Contains("node-b", panel.TextContent);
|
||||
Assert.Contains("OPC-PWD", panel.TextContent);
|
||||
Assert.Contains("did not decrypt on this node", panel.TextContent);
|
||||
Assert.DoesNotContain("node-a", panel.TextContent);
|
||||
});
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(false)] // null = "no data" (central self-report, pre-M1 site)
|
||||
[InlineData(true)] // every node healthy
|
||||
public void DoesNotRender_SecretMirrorWarning_WhenNoNodeIsFailing(bool reported)
|
||||
{
|
||||
SeedStates(new SiteHealthState
|
||||
{
|
||||
SiteId = "site-a",
|
||||
IsOnline = true,
|
||||
LastHeartbeatAt = DateTimeOffset.UtcNow,
|
||||
LatestReport = ReportWithReconciler("site-a", active: true) with
|
||||
{
|
||||
SecretMirrorStatuses = reported
|
||||
? [new SecretMirrorNodeStatus("node-a", DateTimeOffset.UtcNow, [])]
|
||||
: null,
|
||||
},
|
||||
});
|
||||
|
||||
var cut = RenderHealthPage();
|
||||
|
||||
cut.WaitForAssertion(() => Assert.Contains("site-a", cut.Markup));
|
||||
Assert.DoesNotContain("data-test=\"secret-mirror-badge\"", cut.Markup);
|
||||
Assert.DoesNotContain("data-test=\"secret-mirror-failures\"", cut.Markup);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Renders_OfflineSince_ForOfflineSiteWithStatusChange()
|
||||
{
|
||||
|
||||
@@ -369,6 +369,7 @@ public class CentralControlDtoMapperTests
|
||||
Assert.Null(roundTripped.LocalDbReplicationConnected);
|
||||
Assert.Null(roundTripped.LocalDbOplogBacklog);
|
||||
Assert.Null(roundTripped.StartupReconcilerActive);
|
||||
Assert.Null(roundTripped.SecretMirrorStatuses);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -384,10 +385,14 @@ public class CentralControlDtoMapperTests
|
||||
DataConnectionEndpoints = new Dictionary<string, string>(),
|
||||
DataConnectionTagQuality = new Dictionary<string, TagQualityCounts>(),
|
||||
ClusterNodes = [],
|
||||
SecretMirrorStatuses = [],
|
||||
};
|
||||
|
||||
var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original)));
|
||||
|
||||
Assert.NotNull(roundTripped.SecretMirrorStatuses);
|
||||
Assert.Empty(roundTripped.SecretMirrorStatuses);
|
||||
|
||||
Assert.NotNull(roundTripped.DataConnectionEndpoints);
|
||||
Assert.Empty(roundTripped.DataConnectionEndpoints);
|
||||
Assert.NotNull(roundTripped.DataConnectionTagQuality);
|
||||
@@ -559,6 +564,18 @@ public class CentralControlDtoMapperTests
|
||||
DetachedScriptThreads = 2,
|
||||
ScriptRunShedCount = 13,
|
||||
SiteStreamAlarmDropCount = 41L,
|
||||
// Review finding M1: every node's secret mirror status, standby included.
|
||||
SecretMirrorStatuses =
|
||||
[
|
||||
new SecretMirrorNodeStatus("node-a", SampleInstant, []),
|
||||
new SecretMirrorNodeStatus(
|
||||
"node-b",
|
||||
SampleInstant.AddMinutes(-3),
|
||||
[
|
||||
new SecretMirrorFailure("OPC-PWD", "did not decrypt on this node"),
|
||||
new SecretMirrorFailure("MX-KEY", "could not be confirmed applied"),
|
||||
]),
|
||||
],
|
||||
};
|
||||
|
||||
/// <summary>
|
||||
@@ -611,6 +628,17 @@ public class CentralControlDtoMapperTests
|
||||
Assert.Equal(expected.DetachedScriptThreads, actual.DetachedScriptThreads);
|
||||
Assert.Equal(expected.ScriptRunShedCount, actual.ScriptRunShedCount);
|
||||
Assert.Equal(expected.SiteStreamAlarmDropCount, actual.SiteStreamAlarmDropCount);
|
||||
Assert.Equal(expected.SecretMirrorStatuses is null, actual.SecretMirrorStatuses is null);
|
||||
if (expected.SecretMirrorStatuses is not null)
|
||||
{
|
||||
Assert.Equal(expected.SecretMirrorStatuses.Count, actual.SecretMirrorStatuses!.Count);
|
||||
for (var i = 0; i < expected.SecretMirrorStatuses.Count; i++)
|
||||
{
|
||||
Assert.Equal(expected.SecretMirrorStatuses[i].NodeName, actual.SecretMirrorStatuses[i].NodeName);
|
||||
Assert.Equal(expected.SecretMirrorStatuses[i].CheckedAtUtc, actual.SecretMirrorStatuses[i].CheckedAtUtc);
|
||||
Assert.Equal(expected.SecretMirrorStatuses[i].Failures, actual.SecretMirrorStatuses[i].Failures);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
|
||||
+19
@@ -82,4 +82,23 @@ public class SiteCommandDtoMapperSecretsTests
|
||||
var r = new ArtifactDeploymentResponse("dep", "site-a", true, null, DateTimeOffset.UtcNow, applied);
|
||||
Assert.Equal(applied, SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(r)).SecretsApplied);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ArtifactResponse_SecretsRemoved_RoundTrips()
|
||||
{
|
||||
var r = new ArtifactDeploymentResponse(
|
||||
"dep", "site-a", true, null, DateTimeOffset.UtcNow, 0, ["old-1", "old-2"]);
|
||||
Assert.Equal(["old-1", "old-2"], SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(r)).SecretsRemoved!);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ArtifactResponse_NoSecretsRemoved_IsNullOnBothSides()
|
||||
{
|
||||
// Absent on the wire (an old site, or nothing removed) must read back as null.
|
||||
var none = new ArtifactDeploymentResponse("dep", "site-a", true, null, DateTimeOffset.UtcNow, 1);
|
||||
Assert.Null(SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(none)).SecretsRemoved);
|
||||
|
||||
var empty = none with { SecretsRemoved = [] };
|
||||
Assert.Null(SiteCommandDtoMapper.FromProto(SiteCommandDtoMapper.ToProto(empty)).SecretsRemoved);
|
||||
}
|
||||
}
|
||||
|
||||
+178
@@ -1157,6 +1157,184 @@ public class ArtifactDeploymentServiceTests : TestKit
|
||||
/// reconciliation gate can be exercised independently of the wire-level success/failure
|
||||
/// the shared <see cref="ArtifactProbeActor"/> models.
|
||||
/// </summary>
|
||||
// ── Final review M5: a central secret-store outage is contained to the sites that need it ──
|
||||
|
||||
[Fact]
|
||||
public async Task StoreOutage_FailsOnlyTheSecretReferencingSite_AndAuditsIt()
|
||||
{
|
||||
var secretSite = new Site("Secret Site", "sec-site") { Id = 1 };
|
||||
var plainSite = new Site("Plain Site", "plain-site") { Id = 2 };
|
||||
_siteRepo.GetAllSitesAsync(Arg.Any<CancellationToken>()).Returns(new List<Site> { secretSite, plainSite });
|
||||
_siteRepo.GetDataConnectionsBySiteIdAsync(1, Arg.Any<CancellationToken>())
|
||||
.Returns(new List<DataConnection>
|
||||
{
|
||||
new("GW1", "MxGateway", 1) { PrimaryConfiguration = """{"ApiKey":"secret:the-key"}""" }
|
||||
});
|
||||
_siteRepo.GetDataConnectionsBySiteIdAsync(2, Arg.Any<CancellationToken>())
|
||||
.Returns(new List<DataConnection>());
|
||||
_secretStore.GetAsync(Arg.Any<SecretName>(), Arg.Any<CancellationToken>())
|
||||
.Returns<Task<StoredSecret?>>(_ => throw new InvalidOperationException("SQL Server unreachable"));
|
||||
|
||||
var recorder = new ArtifactProbeRecorder();
|
||||
var probe = Sys.ActorOf(Props.Create(() =>
|
||||
new SecretsAckProbeActor(recorder, cmd => cmd.Secrets == null ? (int?)null : cmd.Secrets.Count)));
|
||||
var service = CreateServiceWithCommActor(probe);
|
||||
|
||||
object? audited = null;
|
||||
_audit.When(x => x.LogAsync(
|
||||
Arg.Any<string>(), "DeploySiteSecrets", Arg.Any<string>(), "sec-site", Arg.Any<string>(),
|
||||
Arg.Any<object>(), Arg.Any<CancellationToken>()))
|
||||
.Do(ci => audited = ci[5]);
|
||||
|
||||
// Before M5 the store exception escaped the build loop and the whole sweep threw.
|
||||
var result = await service.DeployToAllSitesAsync("admin");
|
||||
|
||||
Assert.True(result.IsSuccess);
|
||||
Assert.Equal(1, result.Value.SuccessCount);
|
||||
Assert.Equal(1, result.Value.FailureCount);
|
||||
var failed = Assert.Single(result.Value.SiteResults, r => r.SiteId == "sec-site");
|
||||
Assert.False(failed.Success);
|
||||
Assert.Contains("Central secret store could not be read", failed.ErrorMessage);
|
||||
Assert.Contains("SQL Server unreachable", failed.ErrorMessage);
|
||||
Assert.Contains(result.Value.SiteResults, r => r.SiteId == "plain-site" && r.Success);
|
||||
|
||||
// Nothing was sent to the failing site, and it keeps no stamp.
|
||||
var sent = Assert.Single(recorder.Received);
|
||||
Assert.Empty(sent.DataConnections ?? []);
|
||||
Assert.Null(secretSite.ArtifactRevisionHash);
|
||||
Assert.NotNull(plainSite.ArtifactRevisionHash);
|
||||
|
||||
Assert.NotNull(audited);
|
||||
var json = JsonSerializer.Serialize(audited);
|
||||
Assert.Contains("\"success\":false", json);
|
||||
Assert.Contains("Central secret store could not be read", json);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ComputeCurrentHashes_StoreOutage_OmitsOnlyTheSecretReferencingSite()
|
||||
{
|
||||
var secretSite = new Site("Secret Site", "sec-site") { Id = 1 };
|
||||
var plainSite = new Site("Plain Site", "plain-site") { Id = 2 };
|
||||
_siteRepo.GetDataConnectionsBySiteIdAsync(1, Arg.Any<CancellationToken>())
|
||||
.Returns(new List<DataConnection>
|
||||
{
|
||||
new("GW1", "MxGateway", 1) { PrimaryConfiguration = """{"ApiKey":"secret:the-key"}""" }
|
||||
});
|
||||
_siteRepo.GetDataConnectionsBySiteIdAsync(2, Arg.Any<CancellationToken>())
|
||||
.Returns(new List<DataConnection>());
|
||||
_secretStore.GetAsync(Arg.Any<SecretName>(), Arg.Any<CancellationToken>())
|
||||
.Returns<Task<StoredSecret?>>(_ => throw new InvalidOperationException("SQL Server unreachable"));
|
||||
var service = CreateService();
|
||||
|
||||
var hashes = await service.ComputeCurrentArtifactHashesAsync([secretSite, plainSite]);
|
||||
|
||||
// The secret-referencing site is left out (the evaluator reads that as Out of date);
|
||||
// the plain site still gets its hash instead of the whole fleet check failing.
|
||||
Assert.False(hashes.ContainsKey(1));
|
||||
Assert.True(hashes.ContainsKey(2));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task RetryForSite_StoreOutage_FailsWithoutSending()
|
||||
{
|
||||
_siteRepo.GetDataConnectionsBySiteIdAsync(1, Arg.Any<CancellationToken>())
|
||||
.Returns(new List<DataConnection>
|
||||
{
|
||||
new("GW1", "MxGateway", 1) { PrimaryConfiguration = """{"ApiKey":"secret:the-key"}""" }
|
||||
});
|
||||
_secretStore.GetAsync(Arg.Any<SecretName>(), Arg.Any<CancellationToken>())
|
||||
.Returns<Task<StoredSecret?>>(_ => throw new InvalidOperationException("SQL Server unreachable"));
|
||||
|
||||
var recorder = new ArtifactProbeRecorder();
|
||||
var probe = Sys.ActorOf(Props.Create(() => new SecretsAckProbeActor(recorder, _ => 1)));
|
||||
var service = CreateServiceWithCommActor(probe);
|
||||
|
||||
var result = await service.RetryForSiteAsync(1, "sec-site", "admin");
|
||||
|
||||
Assert.True(result.IsFailure);
|
||||
Assert.Contains("Central secret store could not be read", result.Error);
|
||||
Assert.Empty(recorder.Received);
|
||||
await _audit.Received(1).LogAsync(
|
||||
"admin", "DeploySiteSecrets", "SiteSecrets", "sec-site", Arg.Any<string>(),
|
||||
Arg.Any<object>(), Arg.Any<CancellationToken>());
|
||||
}
|
||||
|
||||
// ── Removal-only deploys are audited (the site reports the names it retired) ──
|
||||
|
||||
[Fact]
|
||||
public async Task RemovalOnlyDeploy_WritesAuditRowNamingTheRemovedSecrets()
|
||||
{
|
||||
var site = new Site("Site One", "site-1") { Id = 1 };
|
||||
_siteRepo.GetAllSitesAsync(Arg.Any<CancellationToken>()).Returns(new List<Site> { site });
|
||||
// The site's last secret reference is gone: nothing ships, the site retires "old-key".
|
||||
_siteRepo.GetDataConnectionsBySiteIdAsync(1, Arg.Any<CancellationToken>())
|
||||
.Returns(new List<DataConnection>());
|
||||
|
||||
var recorder = new ArtifactProbeRecorder();
|
||||
var probe = Sys.ActorOf(Props.Create(() => new RemovingProbeActor(recorder, new[] { "old-key" })));
|
||||
var service = CreateServiceWithCommActor(probe);
|
||||
|
||||
object? audited = null;
|
||||
_audit.When(x => x.LogAsync(
|
||||
Arg.Any<string>(), "DeploySiteSecrets", Arg.Any<string>(), "site-1", Arg.Any<string>(),
|
||||
Arg.Any<object>(), Arg.Any<CancellationToken>()))
|
||||
.Do(ci => audited = ci[5]);
|
||||
|
||||
var result = await service.DeployToAllSitesAsync("admin");
|
||||
|
||||
Assert.True(result.IsSuccess);
|
||||
var siteResult = Assert.Single(result.Value.SiteResults);
|
||||
Assert.True(siteResult.Success);
|
||||
Assert.Equal(["old-key"], siteResult.SecretsRemoved!);
|
||||
|
||||
Assert.NotNull(audited);
|
||||
var json = JsonSerializer.Serialize(audited);
|
||||
Assert.Contains("\"removed\":[\"old-key\"]", json);
|
||||
Assert.Contains("\"secrets\":[]", json);
|
||||
Assert.Contains("\"success\":true", json);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task RetryForSite_RemovalOnly_WritesAuditRowNamingTheRemovedSecrets()
|
||||
{
|
||||
_siteRepo.GetDataConnectionsBySiteIdAsync(1, Arg.Any<CancellationToken>())
|
||||
.Returns(new List<DataConnection>());
|
||||
|
||||
var recorder = new ArtifactProbeRecorder();
|
||||
var probe = Sys.ActorOf(Props.Create(() => new RemovingProbeActor(recorder, new[] { "old-key" })));
|
||||
var service = CreateServiceWithCommActor(probe);
|
||||
|
||||
object? audited = null;
|
||||
_audit.When(x => x.LogAsync(
|
||||
Arg.Any<string>(), "DeploySiteSecrets", Arg.Any<string>(), "site-1", Arg.Any<string>(),
|
||||
Arg.Any<object>(), Arg.Any<CancellationToken>()))
|
||||
.Do(ci => audited = ci[5]);
|
||||
|
||||
var result = await service.RetryForSiteAsync(1, "site-1", "admin");
|
||||
|
||||
Assert.True(result.IsSuccess);
|
||||
Assert.NotNull(audited);
|
||||
Assert.Contains("\"removed\":[\"old-key\"]", JsonSerializer.Serialize(audited));
|
||||
}
|
||||
|
||||
/// <summary>Acks every command as applied (0 shipped secrets) and reports the given names as retired.</summary>
|
||||
private class RemovingProbeActor : ReceiveActor
|
||||
{
|
||||
public RemovingProbeActor(ArtifactProbeRecorder recorder, IReadOnlyList<string> removed)
|
||||
{
|
||||
Receive<SiteEnvelope>(env =>
|
||||
{
|
||||
if (env.Message is DeployArtifactsCommand cmd)
|
||||
{
|
||||
recorder.Received.Add(cmd);
|
||||
Sender.Tell(new ArtifactDeploymentResponse(
|
||||
cmd.DeploymentId, env.SiteId, true, null, DateTimeOffset.UtcNow,
|
||||
cmd.Secrets?.Count, removed));
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
private class SecretsAckProbeActor : ReceiveActor
|
||||
{
|
||||
public SecretsAckProbeActor(ArtifactProbeRecorder recorder, Func<DeployArtifactsCommand, int?> secretsApplied)
|
||||
|
||||
@@ -100,6 +100,40 @@ public class SiteSecretsCollectorTests
|
||||
Assert.Empty(result.Missing);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Collect_StoreThrows_ReturnsStoreError_InsteadOfThrowing()
|
||||
{
|
||||
// Final review M5: one site's store read failing must not abort a whole sweep.
|
||||
var store = new FakeSecretStore(Row("a", 1)) { Fault = new InvalidOperationException("db down") };
|
||||
|
||||
var result = await new SiteSecretsCollector(store).CollectAsync(
|
||||
[Dc("GW1", """{"ApiKey":"secret:a"}""")], default);
|
||||
|
||||
Assert.NotNull(result.StoreError);
|
||||
Assert.Contains("'a'", result.StoreError);
|
||||
Assert.Contains("InvalidOperationException: db down", result.StoreError);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Collect_CallerCancellation_StillPropagates()
|
||||
{
|
||||
using var cts = new CancellationTokenSource();
|
||||
cts.Cancel();
|
||||
var store = new FakeSecretStore(Row("a", 1)) { Fault = new OperationCanceledException(cts.Token) };
|
||||
|
||||
await Assert.ThrowsAnyAsync<OperationCanceledException>(() => new SiteSecretsCollector(store).CollectAsync(
|
||||
[Dc("GW1", """{"ApiKey":"secret:a"}""")], cts.Token));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Collect_Success_HasNoStoreError()
|
||||
{
|
||||
var result = await new SiteSecretsCollector(new FakeSecretStore(Row("a", 1))).CollectAsync(
|
||||
[Dc("GW1", """{"ApiKey":"secret:a"}""")], default);
|
||||
|
||||
Assert.Null(result.StoreError);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// In-memory <see cref="ISecretStore"/> stand-in. Implements only <see cref="GetAsync"/> —
|
||||
/// the sole member <see cref="SiteSecretsCollector"/> calls; every other member throws so an
|
||||
@@ -112,8 +146,13 @@ public class SiteSecretsCollectorTests
|
||||
|
||||
public FakeSecretStore(params StoredSecret[] rows) => _rows = rows.ToDictionary(r => r.Name);
|
||||
|
||||
/// <summary>When set, <see cref="GetAsync"/> throws this instead of reading.</summary>
|
||||
public Exception? Fault { get; init; }
|
||||
|
||||
public Task<StoredSecret?> GetAsync(SecretName name, CancellationToken ct) =>
|
||||
Task.FromResult(_rows.TryGetValue(name, out var row) ? row : null);
|
||||
Fault is not null
|
||||
? Task.FromException<StoredSecret?>(Fault)
|
||||
: Task.FromResult(_rows.TryGetValue(name, out var row) ? row : null);
|
||||
|
||||
public Task UpsertAsync(StoredSecret row, CancellationToken ct) => throw new NotSupportedException();
|
||||
|
||||
|
||||
+45
@@ -0,0 +1,45 @@
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Review finding M1 — <c>SiteHealthReport.SecretMirrorStatuses</c> carries every site node's
|
||||
/// deployed-secrets mirror failures (the standby's included) on the active node's report.
|
||||
/// </summary>
|
||||
public class SecretMirrorStatusHealthFieldTests
|
||||
{
|
||||
[Fact]
|
||||
public void CollectReport_NeverTold_LeavesTheFieldNull()
|
||||
{
|
||||
Assert.Null(new SiteHealthCollector().CollectReport("TestSite").SecretMirrorStatuses);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void CollectReport_CarriesTheLatestSnapshot_AndDoesNotDrainIt()
|
||||
{
|
||||
var collector = new SiteHealthCollector();
|
||||
collector.SetSecretMirrorStatuses(
|
||||
[
|
||||
new SecretMirrorNodeStatus("node-a", DateTimeOffset.UtcNow, []),
|
||||
new SecretMirrorNodeStatus(
|
||||
"node-b", DateTimeOffset.UtcNow, [new SecretMirrorFailure("S1", "did not decrypt")]),
|
||||
]);
|
||||
|
||||
var first = collector.CollectReport("TestSite").SecretMirrorStatuses!;
|
||||
var second = collector.CollectReport("TestSite").SecretMirrorStatuses!;
|
||||
|
||||
Assert.Equal(["node-a", "node-b"], first.Select(s => s.NodeName));
|
||||
Assert.Equal(2, second.Count); // point-in-time, not interval-drained
|
||||
Assert.Equal("S1", Assert.Single(first[1].Failures).SecretName);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SetNull_ResetsToNoData()
|
||||
{
|
||||
var collector = new SiteHealthCollector();
|
||||
collector.SetSecretMirrorStatuses([]);
|
||||
collector.SetSecretMirrorStatuses(null);
|
||||
|
||||
Assert.Null(collector.CollectReport("TestSite").SecretMirrorStatuses);
|
||||
}
|
||||
}
|
||||
@@ -212,7 +212,7 @@ public class SiteLocalDbCdcRegistrationTests : IDisposable
|
||||
// ---- helpers ----------------------------------------------------------------------
|
||||
|
||||
/// <summary>
|
||||
/// The eleven replicated tables, plus the two deliberately-unregistered central-only
|
||||
/// The twelve replicated tables, plus the two deliberately-unregistered central-only
|
||||
/// notification tables, plus the node-local (also deliberately-unregistered)
|
||||
/// <c>mirrored_secrets</c> ledger.
|
||||
/// </summary>
|
||||
@@ -221,7 +221,7 @@ public class SiteLocalDbCdcRegistrationTests : IDisposable
|
||||
"OperationTracking", "site_events", "sf_messages", "deployed_configurations",
|
||||
"static_attribute_overrides", "shared_scripts", "external_systems",
|
||||
"database_connections", "data_connection_definitions", "native_alarm_state",
|
||||
"deployed_secrets",
|
||||
"deployed_secrets", "secret_mirror_status",
|
||||
"notification_lists", "smtp_configurations", "mirrored_secrets",
|
||||
];
|
||||
|
||||
@@ -230,7 +230,7 @@ public class SiteLocalDbCdcRegistrationTests : IDisposable
|
||||
"OperationTracking", "site_events", "sf_messages", "deployed_configurations",
|
||||
"static_attribute_overrides", "shared_scripts", "external_systems",
|
||||
"database_connections", "data_connection_definitions", "native_alarm_state",
|
||||
"deployed_secrets",
|
||||
"deployed_secrets", "secret_mirror_status",
|
||||
];
|
||||
|
||||
private static void AssertCaptureTriggersCoverTheReplicatedTables(ILocalDb db)
|
||||
|
||||
@@ -15,8 +15,9 @@ namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
|
||||
/// <c>RegisterReplicated</c> time, which on a site node is a BOOT failure of the replicating
|
||||
/// pair. This test moves that discovery from the first replicated boot to the build: it walks
|
||||
/// the real site schema (every table <c>SiteLocalDbSetup.OnReady</c> creates) and asserts none
|
||||
/// of the eleven <see cref="SiteLocalDbSetup.ReplicatedTables"/> (incl. <c>deployed_secrets</c>,
|
||||
/// added for site-secrets artifact deploy) has such an inbound FK.
|
||||
/// of the twelve <see cref="SiteLocalDbSetup.ReplicatedTables"/> (incl. <c>deployed_secrets</c>,
|
||||
/// added for site-secrets artifact deploy, and <c>secret_mirror_status</c>, added by review
|
||||
/// finding M1) has such an inbound FK.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The schema is built with an UNREPLICATED config on purpose: a replicated one would make
|
||||
|
||||
@@ -254,7 +254,7 @@ public class SiteLocalDbWiringTests : IDisposable
|
||||
"deployed_configurations", "static_attribute_overrides", "shared_scripts",
|
||||
"external_systems", "database_connections", "notification_lists",
|
||||
"data_connection_definitions", "smtp_configurations", "native_alarm_state",
|
||||
"deployed_secrets", "mirrored_secrets",
|
||||
"deployed_secrets", "mirrored_secrets", "secret_mirror_status",
|
||||
})
|
||||
{
|
||||
Assert.Contains(table, tables);
|
||||
@@ -262,9 +262,10 @@ public class SiteLocalDbWiringTests : IDisposable
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Site_LocalDb_RegistersExactlyTheElevenReplicatedTables()
|
||||
public void Site_LocalDb_RegistersExactlyTheTwelveReplicatedTables()
|
||||
{
|
||||
// The Phase 2 cutover, extended by site-secrets artifact deploy's deployed_secrets.
|
||||
// The Phase 2 cutover, extended by site-secrets artifact deploy's deployed_secrets
|
||||
// and review finding M1's per-node secret_mirror_status.
|
||||
// This is an EXACTLY check rather than a Contains in both directions, and both
|
||||
// directions are load-bearing.
|
||||
//
|
||||
@@ -285,8 +286,8 @@ public class SiteLocalDbWiringTests : IDisposable
|
||||
// data_connection_definitions precedes database_connections.
|
||||
"OperationTracking", "data_connection_definitions", "database_connections",
|
||||
"deployed_configurations", "deployed_secrets", "external_systems",
|
||||
"native_alarm_state", "sf_messages", "shared_scripts", "site_events",
|
||||
"static_attribute_overrides",
|
||||
"native_alarm_state", "secret_mirror_status", "sf_messages", "shared_scripts",
|
||||
"site_events", "static_attribute_overrides",
|
||||
],
|
||||
db.ReplicatedTables.Keys.OrderBy(k => k, StringComparer.Ordinal).ToArray());
|
||||
}
|
||||
@@ -334,6 +335,7 @@ public class SiteLocalDbWiringTests : IDisposable
|
||||
["instance_unique_name", "source_canonical_name", "source_reference"],
|
||||
db.ReplicatedTables["native_alarm_state"].PkColumns);
|
||||
Assert.Equal(["name"], db.ReplicatedTables["deployed_secrets"].PkColumns);
|
||||
Assert.Equal(["node_name"], db.ReplicatedTables["secret_mirror_status"].PkColumns);
|
||||
}
|
||||
|
||||
private static HashSet<string> TableNames(ILocalDb db)
|
||||
|
||||
+29
@@ -173,6 +173,35 @@ public class DeploymentManagerActorArtifactSecretsTests : TestKit, IDisposable
|
||||
Assert.Equal(0, response.SecretsApplied);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Secrets_Dropped_ReplyNamesTheRetiredSecrets()
|
||||
{
|
||||
var store = new FakeSecretStore();
|
||||
var resolver = new FakeSecretResolver();
|
||||
var mirror = CreateMirror(store, resolver);
|
||||
var manager = CreateDeploymentManager(mirror);
|
||||
|
||||
manager.Tell(ArtifactsCommand("dep-1", secrets: [Art("a", 1), Art("b", 1)]));
|
||||
var first = ExpectMsg<ArtifactDeploymentResponse>(TimeSpan.FromSeconds(10));
|
||||
Assert.True(first.Success);
|
||||
Assert.Null(first.SecretsRemoved);
|
||||
|
||||
// The next deploy no longer references "b" (a removal-only change for that name):
|
||||
// the reply names it so central can audit the removal.
|
||||
manager.Tell(ArtifactsCommand("dep-2", secrets: [Art("a", 1)]));
|
||||
var second = ExpectMsg<ArtifactDeploymentResponse>(TimeSpan.FromSeconds(10));
|
||||
Assert.True(second.Success);
|
||||
Assert.Equal(["b"], second.SecretsRemoved!);
|
||||
|
||||
// Dropping the last reference retires "a" the same way.
|
||||
manager.Tell(ArtifactsCommand("dep-3", secrets: []));
|
||||
var third = ExpectMsg<ArtifactDeploymentResponse>(TimeSpan.FromSeconds(10));
|
||||
Assert.True(third.Success);
|
||||
Assert.Equal(0, third.SecretsApplied);
|
||||
Assert.Equal(["a"], third.SecretsRemoved!);
|
||||
await Task.CompletedTask;
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Secrets_MirrorFailsOne_SuccessFalse_KekIdInMessage_SecretsAppliedExcludesFailure()
|
||||
{
|
||||
|
||||
+49
@@ -1,6 +1,7 @@
|
||||
using System.Text.Json;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
|
||||
using ZB.MOM.WW.ScadaBridge.TestSupport;
|
||||
|
||||
@@ -303,6 +304,54 @@ public class DeployedSecretsStorageTests : IAsyncLifetime, IDisposable
|
||||
Assert.Equal("A", mirrored.Name);
|
||||
}
|
||||
|
||||
// ── secret_mirror_status (review finding M1) ──
|
||||
|
||||
[Fact]
|
||||
public async Task SecretMirrorStatus_RoundTrips_OneRowPerNode_OrderedByNodeName()
|
||||
{
|
||||
var at = new DateTimeOffset(2026, 9, 28, 6, 0, 0, TimeSpan.Zero);
|
||||
await _storage.SetSecretMirrorStatusAsync(
|
||||
"node-b", [new SecretMirrorFailure("A", "did not decrypt")], at);
|
||||
await _storage.SetSecretMirrorStatusAsync("node-a", [], at.AddMinutes(1));
|
||||
|
||||
var rows = await _storage.GetSecretMirrorStatusesAsync();
|
||||
|
||||
Assert.Equal(["node-a", "node-b"], rows.Select(r => r.NodeName));
|
||||
Assert.Empty(rows[0].Failures);
|
||||
Assert.Equal(at.AddMinutes(1), rows[0].CheckedAtUtc);
|
||||
var failure = Assert.Single(rows[1].Failures);
|
||||
Assert.Equal(new SecretMirrorFailure("A", "did not decrypt"), failure);
|
||||
Assert.Equal(at, rows[1].CheckedAtUtc);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SecretMirrorStatus_Set_UpsertsTheNodesRow()
|
||||
{
|
||||
var at = new DateTimeOffset(2026, 9, 28, 6, 0, 0, TimeSpan.Zero);
|
||||
await _storage.SetSecretMirrorStatusAsync("node-a", [new SecretMirrorFailure("A", "x")], at);
|
||||
await _storage.SetSecretMirrorStatusAsync("node-a", [], at.AddHours(1));
|
||||
|
||||
var row = Assert.Single(await _storage.GetSecretMirrorStatusesAsync());
|
||||
Assert.Empty(row.Failures);
|
||||
Assert.Equal(at.AddHours(1), row.CheckedAtUtc);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SecretMirrorStatus_UnparseableRow_ReadsAsAFailure_NotAsHealthy()
|
||||
{
|
||||
await using (var connection = _storage.CreateConnection())
|
||||
await using (var insert = connection.CreateCommand())
|
||||
{
|
||||
insert.CommandText =
|
||||
"INSERT INTO secret_mirror_status (node_name, failures_json, checked_at) VALUES ('node-b', '{not json', '2026-09-28T00:00:00.0000000+00:00')";
|
||||
await insert.ExecuteNonQueryAsync();
|
||||
}
|
||||
|
||||
var row = Assert.Single(await _storage.GetSecretMirrorStatusesAsync());
|
||||
Assert.Equal("node-b", row.NodeName);
|
||||
Assert.Single(row.Failures);
|
||||
}
|
||||
|
||||
private static SecretArtifact Art(string name, long revision) => new(
|
||||
Name: name,
|
||||
ContentType: "application/octet-stream",
|
||||
|
||||
@@ -23,14 +23,14 @@ public class SiteStorageSchemaTests
|
||||
SiteStorageSchema.Apply(connection);
|
||||
SiteStorageSchema.Apply(connection); // second run must not throw
|
||||
|
||||
// Every table is named explicitly on purpose — a loop asserting "11 tables"
|
||||
// Every table is named explicitly on purpose — a loop asserting "12 tables"
|
||||
// would still pass if one were renamed.
|
||||
foreach (var table in new[]
|
||||
{
|
||||
"deployed_configurations", "static_attribute_overrides", "shared_scripts",
|
||||
"external_systems", "database_connections", "notification_lists",
|
||||
"data_connection_definitions", "smtp_configurations", "native_alarm_state",
|
||||
"deployed_secrets", "mirrored_secrets",
|
||||
"deployed_secrets", "mirrored_secrets", "secret_mirror_status",
|
||||
})
|
||||
{
|
||||
Assert.True(TableExists(connection, table), $"missing table: {table}");
|
||||
|
||||
+131
-2
@@ -2,6 +2,7 @@ using System.Collections.Concurrent;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Secrets;
|
||||
@@ -655,6 +656,132 @@ public class DeployedSecretsMirrorTests : IDisposable
|
||||
}
|
||||
}
|
||||
|
||||
// ── Per-node status row for central (review finding M1) ──
|
||||
|
||||
[Fact]
|
||||
public async Task Status_FirstPassAfterStartup_WritesThisNodesRow_EvenWhenHealthy()
|
||||
{
|
||||
var time = new FakeTimeProvider(new DateTimeOffset(2026, 9, 28, 6, 0, 0, TimeSpan.Zero));
|
||||
var (mirror, storage, _, _, _) = await CreateAsync(time: time, nodeIdentity: new FixedNodeIdentity("node-b"));
|
||||
|
||||
await mirror.ReconcileAsync(CancellationToken.None);
|
||||
|
||||
var status = Assert.Single(await storage.GetSecretMirrorStatusesAsync());
|
||||
Assert.Equal("node-b", status.NodeName);
|
||||
Assert.Empty(status.Failures);
|
||||
Assert.Equal(time.Now, status.CheckedAtUtc);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Status_UnchangedFailureSet_IsNotRewrittenOnLaterPasses()
|
||||
{
|
||||
var time = new FakeTimeProvider(new DateTimeOffset(2026, 9, 28, 6, 0, 0, TimeSpan.Zero));
|
||||
var (mirror, storage, _, resolver, _) = await CreateAsync(time: time, nodeIdentity: new FixedNodeIdentity("node-b"));
|
||||
resolver.ThrowFor("a");
|
||||
await storage.UpsertDeployedSecretsAsync([Art("a", 1, kek: "kek-central")], "d1");
|
||||
|
||||
await mirror.ReconcileAsync(CancellationToken.None);
|
||||
var firstWrite = Assert.Single(await storage.GetSecretMirrorStatusesAsync());
|
||||
|
||||
// Same failure on every later pass: the row is replicated, so it must not be rewritten.
|
||||
time.Now = time.Now.AddMinutes(5);
|
||||
await mirror.ReconcileAsync(CancellationToken.None);
|
||||
await mirror.ReconcileAsync(CancellationToken.None);
|
||||
|
||||
var after = Assert.Single(await storage.GetSecretMirrorStatusesAsync());
|
||||
Assert.Equal(firstWrite.CheckedAtUtc, after.CheckedAtUtc);
|
||||
var failure = Assert.Single(after.Failures);
|
||||
Assert.Equal("a", failure.SecretName);
|
||||
Assert.Contains("kek-central", failure.Message);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Status_FailureSetChange_RewritesTheRow_AndClearsWhenHealed()
|
||||
{
|
||||
var time = new FakeTimeProvider(new DateTimeOffset(2026, 9, 28, 6, 0, 0, TimeSpan.Zero));
|
||||
var (mirror, storage, _, resolver, _) = await CreateAsync(time: time, nodeIdentity: new FixedNodeIdentity("node-a"));
|
||||
await storage.UpsertDeployedSecretsAsync([Art("a", 1)], "d1");
|
||||
|
||||
await mirror.ReconcileAsync(CancellationToken.None);
|
||||
Assert.Empty(Assert.Single(await storage.GetSecretMirrorStatusesAsync()).Failures);
|
||||
|
||||
// A decrypt failure appears -> row rewritten with the failure.
|
||||
resolver.ThrowFor("a");
|
||||
time.Now = time.Now.AddMinutes(1);
|
||||
await mirror.ReconcileAsync(CancellationToken.None);
|
||||
var failing = Assert.Single(await storage.GetSecretMirrorStatusesAsync());
|
||||
Assert.Equal("a", Assert.Single(failing.Failures).SecretName);
|
||||
Assert.Equal(time.Now, failing.CheckedAtUtc);
|
||||
|
||||
// Healed -> row rewritten empty.
|
||||
resolver.ClearFailures();
|
||||
time.Now = time.Now.AddMinutes(1);
|
||||
await mirror.ReconcileAsync(CancellationToken.None);
|
||||
var healed = Assert.Single(await storage.GetSecretMirrorStatusesAsync());
|
||||
Assert.Empty(healed.Failures);
|
||||
Assert.Equal(time.Now, healed.CheckedAtUtc);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Status_RowCarriesNamesAndMessagesOnly_NeverCiphertext()
|
||||
{
|
||||
var (mirror, storage, _, resolver, _) = await CreateAsync(nodeIdentity: new FixedNodeIdentity("node-a"));
|
||||
resolver.ThrowFor("a");
|
||||
await storage.UpsertDeployedSecretsAsync([Art("a", 1)], "d1");
|
||||
|
||||
await mirror.ReconcileAsync(CancellationToken.None);
|
||||
|
||||
await using var connection = storage.CreateConnection();
|
||||
await using var command = connection.CreateCommand();
|
||||
command.CommandText = "SELECT failures_json FROM secret_mirror_status WHERE node_name = 'node-a'";
|
||||
var json = (string)(await command.ExecuteScalarAsync())!;
|
||||
Assert.Contains("\"SecretName\":\"a\"", json);
|
||||
Assert.DoesNotContain("Ciphertext", json);
|
||||
Assert.DoesNotContain(Convert.ToBase64String(new byte[] { 1, 2, 3, 4, 5 }), json);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Status_WriteFailure_DoesNotBreakThePass_AndIsRetriedNextPass()
|
||||
{
|
||||
var logger = new CapturingLogger<DeployedSecretsMirror>();
|
||||
var (mirror, storage, store, _, _) = await CreateAsync(logger: logger, nodeIdentity: new FixedNodeIdentity("node-a"));
|
||||
await storage.UpsertDeployedSecretsAsync([Art("a", 1)], "d1");
|
||||
|
||||
await using (var connection = storage.CreateConnection())
|
||||
await using (var drop = connection.CreateCommand())
|
||||
{
|
||||
drop.CommandText = "DROP TABLE secret_mirror_status";
|
||||
await drop.ExecuteNonQueryAsync();
|
||||
}
|
||||
|
||||
var r = await mirror.ReconcileAsync(CancellationToken.None);
|
||||
|
||||
Assert.Equal(["a"], r.Applied);
|
||||
Assert.True(store.Contains("a"));
|
||||
Assert.Contains(logger.Entries, e => e.Level == LogLevel.Warning && e.Message.Contains("mirror status"));
|
||||
|
||||
// Table back (e.g. transient fault cleared): the unchanged failure set is still
|
||||
// written, because the earlier write never landed.
|
||||
await storage.InitializeAsync();
|
||||
await mirror.ReconcileAsync(CancellationToken.None);
|
||||
Assert.Equal("node-a", Assert.Single(await storage.GetSecretMirrorStatusesAsync()).NodeName);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Status_WithoutNodeIdentity_FallsBackToMachineName()
|
||||
{
|
||||
var (mirror, storage, _, _, _) = await CreateAsync();
|
||||
|
||||
await mirror.ReconcileAsync(CancellationToken.None);
|
||||
|
||||
Assert.Equal(Environment.MachineName, Assert.Single(await storage.GetSecretMirrorStatusesAsync()).NodeName);
|
||||
}
|
||||
|
||||
private sealed class FixedNodeIdentity(string? name) : INodeIdentityProvider
|
||||
{
|
||||
public string? NodeName { get; } = name;
|
||||
}
|
||||
|
||||
// ── Harness ──
|
||||
|
||||
private async Task<(
|
||||
@@ -665,7 +792,8 @@ public class DeployedSecretsMirrorTests : IDisposable
|
||||
FakeSecretCacheInvalidator Invalidator)> CreateAsync(
|
||||
CapturingLogger<DeployedSecretsMirror>? logger = null,
|
||||
TimeSpan? interval = null,
|
||||
TimeProvider? time = null)
|
||||
TimeProvider? time = null,
|
||||
INodeIdentityProvider? nodeIdentity = null)
|
||||
{
|
||||
var storage = new SiteStorageService(_localDb.Db, NullLogger<SiteStorageService>.Instance);
|
||||
await storage.InitializeAsync();
|
||||
@@ -685,7 +813,8 @@ public class DeployedSecretsMirrorTests : IDisposable
|
||||
invalidator,
|
||||
options,
|
||||
(ILogger<DeployedSecretsMirror>?)logger ?? NullLogger<DeployedSecretsMirror>.Instance,
|
||||
time ?? TimeProvider.System);
|
||||
time ?? TimeProvider.System,
|
||||
nodeIdentity);
|
||||
|
||||
return (mirror, storage, store, resolver, invalidator);
|
||||
}
|
||||
|
||||
+95
@@ -0,0 +1,95 @@
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
|
||||
using ZB.MOM.WW.ScadaBridge.HealthMonitoring;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Secrets;
|
||||
using ZB.MOM.WW.ScadaBridge.TestSupport;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Secrets;
|
||||
|
||||
/// <summary>
|
||||
/// Review finding M1: <see cref="SecretMirrorStatusReporter"/> forwards every node's
|
||||
/// replicated <c>secret_mirror_status</c> row onto the site health report via the collector.
|
||||
/// </summary>
|
||||
public class SecretMirrorStatusReporterTests : IAsyncLifetime, IDisposable
|
||||
{
|
||||
private readonly TestLocalDb _localDb = TestLocalDb.CreateTemp("secret-mirror-status");
|
||||
private SiteStorageService _storage = null!;
|
||||
|
||||
public async Task InitializeAsync()
|
||||
{
|
||||
_storage = new SiteStorageService(_localDb.Db, NullLogger<SiteStorageService>.Instance);
|
||||
await _storage.InitializeAsync();
|
||||
}
|
||||
|
||||
public Task DisposeAsync() => Task.CompletedTask;
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
var path = _localDb.Path;
|
||||
_localDb.Dispose();
|
||||
TestLocalDb.DeleteFiles(path);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Probe_PushesEveryNodesRow_OntoTheNextReport()
|
||||
{
|
||||
var at = new DateTimeOffset(2026, 9, 28, 6, 0, 0, TimeSpan.Zero);
|
||||
await _storage.SetSecretMirrorStatusAsync("node-a", [], at);
|
||||
await _storage.SetSecretMirrorStatusAsync(
|
||||
"node-b", [new SecretMirrorFailure("OPC-PWD", "did not decrypt on this node")], at);
|
||||
var collector = new SiteHealthCollector();
|
||||
var reporter = new SecretMirrorStatusReporter(
|
||||
collector, _storage, NullLogger<SecretMirrorStatusReporter>.Instance);
|
||||
|
||||
await reporter.ProbeAsync();
|
||||
var report = collector.CollectReport("site-a");
|
||||
|
||||
Assert.NotNull(report.SecretMirrorStatuses);
|
||||
Assert.Equal(["node-a", "node-b"], report.SecretMirrorStatuses!.Select(s => s.NodeName));
|
||||
Assert.Equal("OPC-PWD", Assert.Single(report.SecretMirrorStatuses[1].Failures).SecretName);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void BeforeAnyProbe_TheReportFieldIsNull()
|
||||
{
|
||||
var report = new SiteHealthCollector().CollectReport("site-a");
|
||||
|
||||
Assert.Null(report.SecretMirrorStatuses);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Probe_NoRowsYet_PushesAnEmptyList_NotNull()
|
||||
{
|
||||
var collector = new SiteHealthCollector();
|
||||
var reporter = new SecretMirrorStatusReporter(
|
||||
collector, _storage, NullLogger<SecretMirrorStatusReporter>.Instance);
|
||||
|
||||
await reporter.ProbeAsync();
|
||||
|
||||
Assert.Empty(collector.CollectReport("site-a").SecretMirrorStatuses!);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Probe_ReadFailure_KeepsThePreviousSnapshot_AndDoesNotThrow()
|
||||
{
|
||||
await _storage.SetSecretMirrorStatusAsync("node-a", [], DateTimeOffset.UtcNow);
|
||||
var collector = new SiteHealthCollector();
|
||||
var reporter = new SecretMirrorStatusReporter(
|
||||
collector, _storage, NullLogger<SecretMirrorStatusReporter>.Instance);
|
||||
await reporter.ProbeAsync();
|
||||
|
||||
await using (var connection = _storage.CreateConnection())
|
||||
await using (var drop = connection.CreateCommand())
|
||||
{
|
||||
drop.CommandText = "DROP TABLE secret_mirror_status";
|
||||
await drop.ExecuteNonQueryAsync();
|
||||
}
|
||||
|
||||
await reporter.ProbeAsync();
|
||||
|
||||
// Point-in-time: not drained on collect, and not cleared by a failed read.
|
||||
Assert.Single(collector.CollectReport("site-a").SecretMirrorStatuses!);
|
||||
Assert.Single(collector.CollectReport("site-a").SecretMirrorStatuses!);
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user