A DeployedSecretsMirror failure on the STANDBY node (e.g. a master-key mismatch
that stops a deployed secret decrypting there) was only logged on that node:
site health reports are sent by the active node only, so central never saw it.
- New 12th LocalDb-replicated table secret_mirror_status (node_name PK,
failures_json of {SecretName, Message}, checked_at; no FKs). Each node's
mirror writes only its own row, and only when its failure set changes plus
once per start-up (every write replicates via CDC). Best-effort: a failed
write is a Warning and is retried next pass. Node key = INodeIdentityProvider
NodeName, falling back to the machine name.
- SecretMirrorStatusReporter (SiteRuntime, every node, 30 s) reads every row
into ISiteHealthCollector.SetSecretMirrorStatuses -> additive nullable
SiteHealthReport.SecretMirrorStatuses (null = no data).
- Wire: SiteHealthReportDto.secret_mirror_statuses = 33 (presence wrapper so
null != empty); vendored CentralControl.cs regenerated; mapper both ways.
- Central UI Health card: 'Secret mirror failing' badge + compact per-node
node / secret / message list.
- Tests: mirror status write-on-change/first-pass/failure isolation, storage
round-trip, reporter/collector, DTO mapper round-trip incl. null/empty,
bUnit, replicated-table set (twelve) + FK guard wording.
Secret-store outage (final review M5): SiteSecretsCollector returns a store
read failure as SiteSecretsCollection.StoreError instead of throwing. A deploy
now fails only the sites that reference a secret (nothing sent, not stamped,
DeploySiteSecrets row with the error), and ComputeCurrentArtifactHashesAsync
omits only those sites (they read Out of date) instead of the exception
aborting the whole sweep and the fleet-wide pending-updates check.
Removal-only deploys: the site reply gains SecretsRemoved (proto field 7,
names only) - the deployed_secrets rows the deploy soft-removed. Central
carries it on SiteArtifactResult and writes the DeploySiteSecrets row with a
removed list whenever it is non-empty, so dropping a site's last secret:
reference is audited centrally rather than only in site logs.
site deploy-artifacts and deploy artifacts rendered the per-site summary and
always exited 0, so a script could not tell a failed site (unreachable,
missing or unreadable secret, partial apply) from a clean deploy. Both now
route through the failureCount rule deploy site already used.
Final review M3: SecretReferenceScanner.TryParseSecretReference (trimmed, case-insensitive
prefix, trimmed name, empty = not a reference, ${secret:} never matches) is used by the scanner
and by MxGatewayDataConnection's ApiKey resolution, so a padded or case-variant reference
resolves the name central ships. A bare 'secret:' ApiKey now fails closed. M7: unit test pins
stamped hash == ComputeCurrentArtifactHashesAsync with a non-empty Secrets list.
Final review M2: HandleDeployArtifacts now sends ApplyArtifactDataConnectionsToDcl after the
shipped secrets are written and mirrored (from a finally, so the push still goes out if the
secrets step fails), so a connection created by the same deploy resolves a newly shipped
secret on first connect. Definitions are still stored at the same point; response semantics
unchanged.
Final review I1: a name already ledgered at the shipped revision is now read back every
pass; a missing, tombstoned or edited local row is healed through the mirror-owned re-stamp
path (bump, apply, read back, invalidate) before verify, instead of a local delete being
misreported as a KEK mismatch that blocked every later deploy. Drift logs Warning once per
occurrence. M4: row_json that cannot be parsed becomes a Failed entry for that name via the
new SiteStorageService.GetDeployedSecretsTolerantAsync; an unreadable removed row is still
honoured by name. M10: invalidator ctor doc corrected.
The site applies DeployArtifactsCommand.Secrets as the full set (rows absent from it are
soft-removed and DeployedSecretsMirror deletes them on both nodes), while null means
'central shipped no secrets' and touches nothing. The command builder shipped null for a
site with no references, so dropping the LAST secret: reference never removed the secret
from either site node. Found by the Task 13 live gate (check 5: site-a-a and site-a-b
both still held sb-gate-mxgw r1, deployed_secrets.is_removed=0 after the removal deploy).
Central now always ships a list. Old sites ignore proto field 9; the artifact hash is
unchanged (empty and null hash identically); SecretsDeployed stays null and no
DeploySiteSecrets audit row is written when nothing shipped. Regression test
NoSecretReferences_ShipsEmptySecretsList_SoTheSiteRemovesPreviouslyDeployedOnes (red
before, green after). Spec notes in Component-DeploymentManager/SiteRuntime + the design
doc's as-built section.
Task 6 appended SiteSecretsCollector to ArtifactDeploymentService's constructor; the
NSubstitute class proxy in ManagementActorTests still passed the old ten arguments, so
the four DeployArtifacts_* tests failed with 'Could not find a constructor'. Caught by
the Task 13 full-solution run.
- SecretsRegistration: UsesCentralSharedStore predicate shared by AddScadaBridgeSecrets and the
new AddPreHostExpander, so a Central node's Layer-A ${secret:} expander composes against the
shared SQL store whenever Secrets:Store:Provider=SqlServer does — closing the gap where a
central migrated off the retired Grpc topology would boot-time-resolve config from per-node
SQLite while serving the shared store.
- Program.cs: expander block now calls AddPreHostExpander + resolves the store-agnostic
ISecretsStoreMigrator seam; stale Grpc-hub/replication comment rewritten.
- DeployedSecretsMirror.cs: stale "gRPC-hub follower half" doc comment rewritten to state the
now-permanent local-SQLite-only invariant.
- StartupValidator: Secrets:Replication:Enabled read as a raw string (any non-empty, non-"false"
value rejected) instead of GetValue<bool>, which threw a raw binder crash on "yes"/"1"; added a
CollectWarnings entry for leftover Secrets:Replication/Secrets:GrpcHub child keys once Enabled
is false/unset.
- Tests: +15 (AddPreHostExpander/UsesCentralSharedStore wiring, Replication:Enabled edge values,
leftover-key warnings). Full solution build and full Host.Tests green (593/593).
Fix round 2 (review task-9-rereview-1.md), both Minors:
1. Clock skew. ApplyAndConfirmAsync now decides the tombstone-vs-out-of-band split by LEDGER
OWNERSHIP (any mirrored_secrets entry for the name, at any revision), not merely by whether
the local row happens to be a tombstone. A live local row this node already owns but whose
ciphertext no longer matches the newly shipped revision (e.g. an earlier tombstone-recovery
re-stamp landed ahead of a since-shipped central UpdatedUtc, under site/central clock skew)
is now re-stamped and re-applied exactly like a tombstone, instead of being misreported as
someone else's "newer local value" and left permanently un-converged. The out-of-band Failed
message is now only ever produced for a name that is NOT in the ledger. New test:
Reconcile_ClockSkewedRecoveredRow_StillConvergesOnNextCentralRevision (a skewed future
re-stamp followed by a genuinely new central revision converges and re-invalidates the
cache).
2. Real-cipher integration test. Added DeployedSecretsMirrorRealStackTests, running the mirror
against the REAL ZB.MOM.WW.Secrets stack (SqliteSecretStore + AesGcmEnvelopeCipher +
DefaultSecretResolver over a real temp-file SQLite database, plus a hand-written
IMasterKeyProvider test double mirroring the library's own) instead of in-memory fakes:
encrypt -> SecretArtifact -> apply -> delete (tombstone) -> re-add the same artifact -> the
recovered row resolves to the original plaintext. Pins the "UpdatedUtc/Revision are not
AEAD-bound" property against the real cipher rather than by source inspection alone.
Test-only package reference: ZB.MOM.WW.Secrets (SiteRuntime itself still depends only on
ZB.MOM.WW.Secrets.Abstractions).
Verify: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter FullyQualifiedName~DeployedSecretsMirrorTests (25/25); full SiteRuntime.Tests suite 878/878.
Fix round 1 (review task-9-review.md), all rulings:
1. (Important 1) ApplyReplicatedAsync is no longer trusted blindly. Every apply is
confirmed by a store.GetAsync read-back before ledgering: a tombstoned local row
(routine remove-then-re-add) is safely re-stamped past the tombstone and re-applied
(confirmed neither UpdatedUtc nor Revision is AEAD-bound in AesGcmEnvelopeCipher, so
the bump cannot break decryption); a live local row with different ciphertext (a
strictly-newer out-of-band secret) is left untouched, never ledgered, never verified
against, and reported with a distinct Failed message so a later central tombstone can
never delete it via this mirror.
2. (Important 2) Per-row try/catch (rethrow only on ct.IsCancellationRequested) so one
poisoned row (e.g. an unrecognised ContentType from a newer central) no longer blocks
every row after it.
3. (Important 3) Optional ISecretCacheInvalidator, invalidated after every successful
apply/delete and before verify, so a rotated/removed secret's plaintext cannot be
served stale for the remainder of the resolver's TTL.
4. (Minor 4) Loop and VerifyAsync OperationCanceledException handling now filters on
ct/stoppingToken.IsCancellationRequested specifically, rather than swallowing every OCE.
5. (Minor 5) Class remarks note the mirror assumes a node-local ISecretStore; Task 11
will reject SqlServer secrets replication on the Site role.
6. (Minor 6/7) Added cancellation + concurrent-callers tests, dropped the redundant
FailFor (tests now use ThrowFor/ReturnNullFor to exercise distinct decrypt-failure
paths), and Dispose() now disposes the gate SemaphoreSlim.
Verify: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter FullyQualifiedName~DeployedSecretsMirrorTests (24/24); full SiteRuntime.Tests suite 876/876.
ArtifactDeploymentService now resolves each site's secret: references (via
the new SiteSecretsCollector, wrapping ISecretStore) into DeployArtifactsCommand.Secrets,
never plaintext. A referenced secret that is absent or tombstoned fails
that site's deploy before anything is sent, with a named error, while the
rest of the batch still deploys; ComputeCurrentArtifactHashesAsync instead
folds a MissingSecrets marker into the hash so such a site reads Out of
date. Central logs what it ships (name@revision) and errors on what's
missing, never material.
Also updates the two other direct ArtifactDeploymentService construction
sites (PendingUpdatesEvaluatorTests, CentralUI SitesPageTests) for the new
constructor parameter.
Script file access (design docs/plans/2026-09-28-script-file-access-design.md): Files.ReadTextAsync /
ReadLinesAsync / ExistsAsync on site ScriptGlobals (+ Instance.Files), backed by the site-singleton
ScriptFileReader. The one host-mediated exception to the System.IO ban - scripts still never name a
System.IO type. OFF by default: ScadaBridge:SiteRuntime:ScriptFileAccess:AllowedRoots (empty = every call
throws Disabled) + MaxFileBytes (1 MiB, validated 1..64 MiB, roots must be absolute). Guard: fully
qualified, no device paths / ADS, canonical path strictly inside a root (directory-boundary match), no
reparse point on the file or any directory below the root, size cap before and during the read.
Errors are one Commons type, ScriptFileAccessException(+ScriptFileAccessFailure); Commons.Types.Scripts
joins the default script imports on every surface so it binds by short name. Mirrored on
ScriptCompileSurface (CompileFiles, parity-pinned) and the Test Run sandbox (binds, refuses Disabled).
Inbound (central) scripts do not get it.
Rig: docker/recipes (sample RecipeValueName,RecipeValue files incl. an invalid one) bind-mounted
read-only at /app/recipes on both site-a nodes with AllowedRoots [/app/recipes]. The TestMesMachine
SimDelmiaRecipe simulator now reads + parses the downloaded file, converts each value to its recipe tag's
type and loads it all-or-nothing (14 new TestDelmiaReceiver attributes); verify-mes-test-machine.sh
group A covers accept / re-download / invalid lines / NotFound / OutsideAllowedRoots, VERIFY_GROUPS
selects groups (rig: AD).
OnParametersSetAsync awaits the principal. On a real cookie session that await
suspends, so the FIRST render ran with _visible still false and
OnAfterRenderAsync(firstRender: true) skipped EnsurePollingStarted(); the
render that followed the resolve was firstRender: false, and nothing else
started the poll unless AuthenticationStateChanged happened to fire. The block
therefore rendered and sat on 'Checking for updates…' for the life of the
circuit. OnParametersSetAsync now starts polling itself when visible (the
start is idempotent and the OnAfterRender call is kept).
Also corrects the comments that called the non-forced
GetAuthenticationStateAsync cheap: on CookieAuthenticationStateProvider both
entry points revalidate against the database and differ only in failClosed, so
the saving is not asking the provider on a tick and reusing the cascaded state
the router already resolved. The provider fallback is a real refresh, is only
taken when no cascade is present (AddCascadingAuthenticationState is
registered, so it is dead in production), and is now resolved at most once per
auth state rather than on every parameter set.
Claude-Session: https://claude.ai/code/session_01Y5ef34iUUMjmwnMpL9xgmB
The gate could pass vacuously. Every site-list read went through a bare
`ROWS=$(site_rows)` with neither the exit status nor the row count checked, and
an empty result satisfied every downstream assertion for free: no rows means no
malformed hash for B1/B2 to report, and a diff of two empty strings is no
difference at all for C1/D2. A down management API would therefore have read as
a clean gate.
Reads now go through read_rows, which fails loudly on a non-zero CLI exit
(site_rows exits 2 on an empty list, and pipefail surfaces a failed call) and
aborts unless the line count equals the site count established in step A —
keeping the running tally so the abort still prints PASS/FAIL. Verified against
an unreachable URL: FAIL 1, exit 1.
The join(1)/cut/diff comparisons assumed lexicographic id order matches numeric
order, which breaks once a site id reaches 10 alongside single-digit ids (and BSD
join silently drops the unmatched rows rather than complaining). Replaced with a
python3 comparison keyed by integer id that also asserts the id SETS match, so a
site appearing or vanishing between two reads invalidates the comparison instead
of quietly shrinking it. Same for the summary table.
Also: set -euo pipefail like every sibling script (the cmd && pass || fail guards
keep -e safe, and -e does not cover var=$(cmd) — hence the explicit checks
above); the cleanup trap is armed BEFORE the shared-script create and falls back
to finding the probe by its unique name, so a create that succeeds server-side
but whose id we fail to parse can no longer orphan a probe; mktemp instead of
/tmp/pu-deploy.$$.
PendingUpdatesTests: the rail-scroll check asserted scrollHeight >= clientHeight,
which holds for every element. It now asserts what the pane is actually for —
the pane is bounded by the viewport and the rail footer holding the indicator
stays on screen, so a nav that grows the rail instead of scrolling fails. Live
values are genuinely overflowing (scrollHeight 1196 vs clientHeight 680, footer
bottom 765 of 900), so the assertion has real content.
Re-run against the rig: PASS 13 / FAIL 0; Playwright 3/3.
Claude-Session: https://claude.ai/code/session_01Y5ef34iUUMjmwnMpL9xgmB
SiteScopeService's instance methods resolve the AUTHORITATIVE state — a
fail-closed, DB-backed group-to-role re-mapping. The indicator called them on
every 30s tick, and it lives in MainLayout, so that was one or two forced
session revalidations per open circuit per page, forever: exactly the
per-circuit cost IPendingUpdatesCache exists to avoid.
Extracts the pure rule as SiteScopeService.Resolve(ClaimsPrincipal) (no I/O;
ResolveAsync now delegates to it, no behaviour change for existing callers).
The indicator takes the cascaded AuthenticationState — falling back to a
NON-forced GetAuthenticationStateAsync, as Templates/FlowDebugger do — and
computes roles plus scope once per auth state, re-running on
AuthenticationStateChanged so a mid-session revocation hides the block
instead of leaving it stale for the life of the circuit. Also routes the
first load through PollGate, adds aria-hidden to the icons, and comments the
off-renderer field writes and the InvalidOperationException catch.
Claude-Session: https://claude.ai/code/session_01Y5ef34iUUMjmwnMpL9xgmB
Rig rebuilt from this branch (image built at d8979664; migration
20260914173147_AddSiteArtifactRevision applied by central-b on boot).
verify-pending-updates.sh drives the CLI end to end: the site list carries the
artifact-revision fields, a fleet-wide deploy stamps every site with a
sha256:<64 hex> hash and a fresh timestamp, creating a shared script leaves the
stamp alone until the next deploy and then moves EVERY site's hash, and deleting
it again returns every hash byte-for-byte to its pre-create value — so the hash
is content-addressed, not deploy-addressed. PASS 13 / FAIL 0.
Step A asserts the SHAPE rather than key presence: the management API serializer
omits null properties, so on a never-deployed site the two keys are absent, not
present-and-null. Step E is a soft reachability check only; the badge text and
the rail indicator land after the Blazor circuit's post-render load.
PendingUpdatesTests covers that browser half — the rail footer settles off
"Checking for updates…", the Sites page carries an Artifacts badge per site card
plus a working Re-check control, and the rail nav's scroll pane computes
overflow-y: auto at lg+. 3/3 passed against the rig (rail read "Up to date",
3 badges over 3 cards, all "Artifacts current").
Claude-Session: https://claude.ai/code/session_01Y5ef34iUUMjmwnMpL9xgmB
The forced refresh after a successful Deploy Artifacts ran inside the try, so
the Bulk-actions toggle and every per-site Deploy item stayed disabled for a
whole fleet evaluation after the success toast; it now runs after the finally,
behind a re-render, with the Re-check spinner showing it in flight. Also:
pluralise the instance line's noun, drop a stale "as of" caption when a
re-check faults, skip the fleet evaluation when the site load itself failed,
and render the badge tooltip through UtcDateTime so the UTC label is honest.
Claude-Session: https://claude.ai/code/session_01Y5ef34iUUMjmwnMpL9xgmB
Adds PendingUpdatesIndicator to MainLayout's RailFooter, above .rail-user.
Self-gating on the Deployer/Administrator role claims, it polls the
process-wide single-flight IPendingUpdatesCache every 30s through PollGate
and renders four states: checking / up to date / pending (warning pill plus
an 'N instances · M sites' breakdown) / unavailable. Each half of the
breakdown links only for the role that can act on it (Topology => Deployer,
Sites => Administrator), and a site-scoped operator's totals are narrowed
through SiteScopeService so they never count a site they cannot see.
Claude-Session: https://claude.ai/code/session_01Y5ef34iUUMjmwnMpL9xgmB
Each site card carries an Artifacts badge (current / out of date / never
deployed) plus an "N of M instances out of date" line linking to Topology,
both read from the shared pending-updates snapshot. The snapshot loads after
the site list so a slow fleet evaluation never holds the cards, and a failure
degrades the badge to "unavailable" instead of taking the page down. Header
gains a Re-check button and an as-of caption; a successful Deploy Artifacts
forces a refresh so the badge flips immediately.
Claude-Session: https://claude.ai/code/session_01Y5ef34iUUMjmwnMpL9xgmB
Both repositories are scoped over one DbContext, so a failed stamp save left the
Site entries tracked as Modified and the deployment record's (uncaught)
SaveChangesAsync rethrew it — losing the audit row and failing a deploy in which
every site succeeded. Ordering rule now: audit row first, stamp after, inside its
own try/catch. Same in RetryForSiteAsync, where the site load is caught too so
nothing on the stamp path can reach the method's catch.
Also: the method-definitions canonicaliser matches the name property
case-insensitively (a naming-policy change at the producer would otherwise
silently turn the sort into a no-op), and ComputeCurrentArtifactHashesAsync
documents that a per-site query failure aborts the whole batch by design.
Claude-Session: https://claude.ai/code/session_01Y5ef34iUUMjmwnMpL9xgmB
One scoped read that answers "is anything waiting to be deployed?" for the whole
fleet: each site's stored artifact revision hash compared against what a deploy
would send right now (NeverDeployed / Current / OutOfDate), plus per-site
deployed/stale/unknown instance tallies from the batched staleness evaluation.
NeverDeployed is deliberately NOT a pending update — the operator has not deployed
yet — and exceptions propagate so the caller's single-flight cache can render
"unavailable" rather than invent an "up to date". Adds the validated
PendingUpdatesCacheTtl (60 s) that TTL will key off.
Claude-Session: https://claude.ai/code/session_01Y5ef34iUUMjmwnMpL9xgmB
ArtifactRevisionHashService (deterministic SHA-256 of the per-site
DeployArtifactsCommand, DeploymentId/timestamp excluded) stamps
Site.ArtifactRevisionHash/ArtifactsDeployedAt for every site whose deploy
succeeded; failed sites keep their previous stamp.
ComputeCurrentArtifactHashesAsync answers "what would it be now".
Claude-Session: https://claude.ai/code/session_01Y5ef34iUUMjmwnMpL9xgmB
DeploymentService.GetInstanceStalenessAsync shares a FlattenSession across
the batch, reads only the (InstanceId, RevisionHash) projection and pays the
snapshot read + I-1 List normalisation only for a mismatch, so the fleet
aggregate agrees exactly with the per-instance Topology badge. The I-1 logic
is extracted from GetDeploymentComparisonAsync unchanged.
Claude-Session: https://claude.ai/code/session_01Y5ef34iUUMjmwnMpL9xgmB
Site.ArtifactRevisionHash / ArtifactsDeployedAt (nullable, migration
AddSiteArtifactRevision) so the Sites page can tell whether a site's
system-wide artifacts are current. Not transported. Adds the
GetDeployedRevisionHashesAsync projection for the fleet-wide staleness batch.
Claude-Session: https://claude.ai/code/session_01Y5ef34iUUMjmwnMpL9xgmB
Joseph Doherty