A DeployedSecretsMirror failure on the STANDBY node (e.g. a master-key mismatch
that stops a deployed secret decrypting there) was only logged on that node:
site health reports are sent by the active node only, so central never saw it.
- New 12th LocalDb-replicated table secret_mirror_status (node_name PK,
failures_json of {SecretName, Message}, checked_at; no FKs). Each node's
mirror writes only its own row, and only when its failure set changes plus
once per start-up (every write replicates via CDC). Best-effort: a failed
write is a Warning and is retried next pass. Node key = INodeIdentityProvider
NodeName, falling back to the machine name.
- SecretMirrorStatusReporter (SiteRuntime, every node, 30 s) reads every row
into ISiteHealthCollector.SetSecretMirrorStatuses -> additive nullable
SiteHealthReport.SecretMirrorStatuses (null = no data).
- Wire: SiteHealthReportDto.secret_mirror_statuses = 33 (presence wrapper so
null != empty); vendored CentralControl.cs regenerated; mapper both ways.
- Central UI Health card: 'Secret mirror failing' badge + compact per-node
node / secret / message list.
- Tests: mirror status write-on-change/first-pass/failure isolation, storage
round-trip, reporter/collector, DTO mapper round-trip incl. null/empty,
bUnit, replicated-table set (twelve) + FK guard wording.
Final review M3: SecretReferenceScanner.TryParseSecretReference (trimmed, case-insensitive
prefix, trimmed name, empty = not a reference, ${secret:} never matches) is used by the scanner
and by MxGatewayDataConnection's ApiKey resolution, so a padded or case-variant reference
resolves the name central ships. A bare 'secret:' ApiKey now fails closed. M7: unit test pins
stamped hash == ComputeCurrentArtifactHashesAsync with a non-empty Secrets list.
Final review M2: HandleDeployArtifacts now sends ApplyArtifactDataConnectionsToDcl after the
shipped secrets are written and mirrored (from a finally, so the push still goes out if the
secrets step fails), so a connection created by the same deploy resolves a newly shipped
secret on first connect. Definitions are still stored at the same point; response semantics
unchanged.
Final review I1: a name already ledgered at the shipped revision is now read back every
pass; a missing, tombstoned or edited local row is healed through the mirror-owned re-stamp
path (bump, apply, read back, invalidate) before verify, instead of a local delete being
misreported as a KEK mismatch that blocked every later deploy. Drift logs Warning once per
occurrence. M4: row_json that cannot be parsed becomes a Failed entry for that name via the
new SiteStorageService.GetDeployedSecretsTolerantAsync; an unreadable removed row is still
honoured by name. M10: invalidator ctor doc corrected.
The site applies DeployArtifactsCommand.Secrets as the full set (rows absent from it are
soft-removed and DeployedSecretsMirror deletes them on both nodes), while null means
'central shipped no secrets' and touches nothing. The command builder shipped null for a
site with no references, so dropping the LAST secret: reference never removed the secret
from either site node. Found by the Task 13 live gate (check 5: site-a-a and site-a-b
both still held sb-gate-mxgw r1, deployed_secrets.is_removed=0 after the removal deploy).
Central now always ships a list. Old sites ignore proto field 9; the artifact hash is
unchanged (empty and null hash identically); SecretsDeployed stays null and no
DeploySiteSecrets audit row is written when nothing shipped. Regression test
NoSecretReferences_ShipsEmptySecretsList_SoTheSiteRemovesPreviouslyDeployedOnes (red
before, green after). Spec notes in Component-DeploymentManager/SiteRuntime + the design
doc's as-built section.
- SecretsRegistration: UsesCentralSharedStore predicate shared by AddScadaBridgeSecrets and the
new AddPreHostExpander, so a Central node's Layer-A ${secret:} expander composes against the
shared SQL store whenever Secrets:Store:Provider=SqlServer does — closing the gap where a
central migrated off the retired Grpc topology would boot-time-resolve config from per-node
SQLite while serving the shared store.
- Program.cs: expander block now calls AddPreHostExpander + resolves the store-agnostic
ISecretsStoreMigrator seam; stale Grpc-hub/replication comment rewritten.
- DeployedSecretsMirror.cs: stale "gRPC-hub follower half" doc comment rewritten to state the
now-permanent local-SQLite-only invariant.
- StartupValidator: Secrets:Replication:Enabled read as a raw string (any non-empty, non-"false"
value rejected) instead of GetValue<bool>, which threw a raw binder crash on "yes"/"1"; added a
CollectWarnings entry for leftover Secrets:Replication/Secrets:GrpcHub child keys once Enabled
is false/unset.
- Tests: +15 (AddPreHostExpander/UsesCentralSharedStore wiring, Replication:Enabled edge values,
leftover-key warnings). Full solution build and full Host.Tests green (593/593).
Fix round 2 (review task-9-rereview-1.md), both Minors:
1. Clock skew. ApplyAndConfirmAsync now decides the tombstone-vs-out-of-band split by LEDGER
OWNERSHIP (any mirrored_secrets entry for the name, at any revision), not merely by whether
the local row happens to be a tombstone. A live local row this node already owns but whose
ciphertext no longer matches the newly shipped revision (e.g. an earlier tombstone-recovery
re-stamp landed ahead of a since-shipped central UpdatedUtc, under site/central clock skew)
is now re-stamped and re-applied exactly like a tombstone, instead of being misreported as
someone else's "newer local value" and left permanently un-converged. The out-of-band Failed
message is now only ever produced for a name that is NOT in the ledger. New test:
Reconcile_ClockSkewedRecoveredRow_StillConvergesOnNextCentralRevision (a skewed future
re-stamp followed by a genuinely new central revision converges and re-invalidates the
cache).
2. Real-cipher integration test. Added DeployedSecretsMirrorRealStackTests, running the mirror
against the REAL ZB.MOM.WW.Secrets stack (SqliteSecretStore + AesGcmEnvelopeCipher +
DefaultSecretResolver over a real temp-file SQLite database, plus a hand-written
IMasterKeyProvider test double mirroring the library's own) instead of in-memory fakes:
encrypt -> SecretArtifact -> apply -> delete (tombstone) -> re-add the same artifact -> the
recovered row resolves to the original plaintext. Pins the "UpdatedUtc/Revision are not
AEAD-bound" property against the real cipher rather than by source inspection alone.
Test-only package reference: ZB.MOM.WW.Secrets (SiteRuntime itself still depends only on
ZB.MOM.WW.Secrets.Abstractions).
Verify: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter FullyQualifiedName~DeployedSecretsMirrorTests (25/25); full SiteRuntime.Tests suite 878/878.
Fix round 1 (review task-9-review.md), all rulings:
1. (Important 1) ApplyReplicatedAsync is no longer trusted blindly. Every apply is
confirmed by a store.GetAsync read-back before ledgering: a tombstoned local row
(routine remove-then-re-add) is safely re-stamped past the tombstone and re-applied
(confirmed neither UpdatedUtc nor Revision is AEAD-bound in AesGcmEnvelopeCipher, so
the bump cannot break decryption); a live local row with different ciphertext (a
strictly-newer out-of-band secret) is left untouched, never ledgered, never verified
against, and reported with a distinct Failed message so a later central tombstone can
never delete it via this mirror.
2. (Important 2) Per-row try/catch (rethrow only on ct.IsCancellationRequested) so one
poisoned row (e.g. an unrecognised ContentType from a newer central) no longer blocks
every row after it.
3. (Important 3) Optional ISecretCacheInvalidator, invalidated after every successful
apply/delete and before verify, so a rotated/removed secret's plaintext cannot be
served stale for the remainder of the resolver's TTL.
4. (Minor 4) Loop and VerifyAsync OperationCanceledException handling now filters on
ct/stoppingToken.IsCancellationRequested specifically, rather than swallowing every OCE.
5. (Minor 5) Class remarks note the mirror assumes a node-local ISecretStore; Task 11
will reject SqlServer secrets replication on the Site role.
6. (Minor 6/7) Added cancellation + concurrent-callers tests, dropped the redundant
FailFor (tests now use ThrowFor/ReturnNullFor to exercise distinct decrypt-failure
paths), and Dispose() now disposes the gate SemaphoreSlim.
Verify: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter FullyQualifiedName~DeployedSecretsMirrorTests (24/24); full SiteRuntime.Tests suite 876/876.
ArtifactDeploymentService now resolves each site's secret: references (via
the new SiteSecretsCollector, wrapping ISecretStore) into DeployArtifactsCommand.Secrets,
never plaintext. A referenced secret that is absent or tombstoned fails
that site's deploy before anything is sent, with a named error, while the
rest of the batch still deploys; ComputeCurrentArtifactHashesAsync instead
folds a MissingSecrets marker into the hash so such a site reads Out of
date. Central logs what it ships (name@revision) and errors on what's
missing, never material.
Also updates the two other direct ArtifactDeploymentService construction
sites (PendingUpdatesEvaluatorTests, CentralUI SitesPageTests) for the new
constructor parameter.
Script file access (design docs/plans/2026-09-28-script-file-access-design.md): Files.ReadTextAsync /
ReadLinesAsync / ExistsAsync on site ScriptGlobals (+ Instance.Files), backed by the site-singleton
ScriptFileReader. The one host-mediated exception to the System.IO ban - scripts still never name a
System.IO type. OFF by default: ScadaBridge:SiteRuntime:ScriptFileAccess:AllowedRoots (empty = every call
throws Disabled) + MaxFileBytes (1 MiB, validated 1..64 MiB, roots must be absolute). Guard: fully
qualified, no device paths / ADS, canonical path strictly inside a root (directory-boundary match), no
reparse point on the file or any directory below the root, size cap before and during the read.
Errors are one Commons type, ScriptFileAccessException(+ScriptFileAccessFailure); Commons.Types.Scripts
joins the default script imports on every surface so it binds by short name. Mirrored on
ScriptCompileSurface (CompileFiles, parity-pinned) and the Test Run sandbox (binds, refuses Disabled).
Inbound (central) scripts do not get it.
Rig: docker/recipes (sample RecipeValueName,RecipeValue files incl. an invalid one) bind-mounted
read-only at /app/recipes on both site-a nodes with AllowedRoots [/app/recipes]. The TestMesMachine
SimDelmiaRecipe simulator now reads + parses the downloaded file, converts each value to its recipe tag's
type and loads it all-or-nothing (14 new TestDelmiaReceiver attributes); verify-mes-test-machine.sh
group A covers accept / re-download / invalid lines / NotFound / OutsideAllowedRoots, VERIFY_GROUPS
selects groups (rig: AD).
OnParametersSetAsync awaits the principal. On a real cookie session that await
suspends, so the FIRST render ran with _visible still false and
OnAfterRenderAsync(firstRender: true) skipped EnsurePollingStarted(); the
render that followed the resolve was firstRender: false, and nothing else
started the poll unless AuthenticationStateChanged happened to fire. The block
therefore rendered and sat on 'Checking for updates…' for the life of the
circuit. OnParametersSetAsync now starts polling itself when visible (the
start is idempotent and the OnAfterRender call is kept).
Also corrects the comments that called the non-forced
GetAuthenticationStateAsync cheap: on CookieAuthenticationStateProvider both
entry points revalidate against the database and differ only in failClosed, so
the saving is not asking the provider on a tick and reusing the cascaded state
the router already resolved. The provider fallback is a real refresh, is only
taken when no cascade is present (AddCascadingAuthenticationState is
registered, so it is dead in production), and is now resolved at most once per
auth state rather than on every parameter set.
Claude-Session: https://claude.ai/code/session_01Y5ef34iUUMjmwnMpL9xgmB
SiteScopeService's instance methods resolve the AUTHORITATIVE state — a
fail-closed, DB-backed group-to-role re-mapping. The indicator called them on
every 30s tick, and it lives in MainLayout, so that was one or two forced
session revalidations per open circuit per page, forever: exactly the
per-circuit cost IPendingUpdatesCache exists to avoid.
Extracts the pure rule as SiteScopeService.Resolve(ClaimsPrincipal) (no I/O;
ResolveAsync now delegates to it, no behaviour change for existing callers).
The indicator takes the cascaded AuthenticationState — falling back to a
NON-forced GetAuthenticationStateAsync, as Templates/FlowDebugger do — and
computes roles plus scope once per auth state, re-running on
AuthenticationStateChanged so a mid-session revocation hides the block
instead of leaving it stale for the life of the circuit. Also routes the
first load through PollGate, adds aria-hidden to the icons, and comments the
off-renderer field writes and the InvalidOperationException catch.
Claude-Session: https://claude.ai/code/session_01Y5ef34iUUMjmwnMpL9xgmB
The forced refresh after a successful Deploy Artifacts ran inside the try, so
the Bulk-actions toggle and every per-site Deploy item stayed disabled for a
whole fleet evaluation after the success toast; it now runs after the finally,
behind a re-render, with the Re-check spinner showing it in flight. Also:
pluralise the instance line's noun, drop a stale "as of" caption when a
re-check faults, skip the fleet evaluation when the site load itself failed,
and render the badge tooltip through UtcDateTime so the UTC label is honest.
Claude-Session: https://claude.ai/code/session_01Y5ef34iUUMjmwnMpL9xgmB
Adds PendingUpdatesIndicator to MainLayout's RailFooter, above .rail-user.
Self-gating on the Deployer/Administrator role claims, it polls the
process-wide single-flight IPendingUpdatesCache every 30s through PollGate
and renders four states: checking / up to date / pending (warning pill plus
an 'N instances · M sites' breakdown) / unavailable. Each half of the
breakdown links only for the role that can act on it (Topology => Deployer,
Sites => Administrator), and a site-scoped operator's totals are narrowed
through SiteScopeService so they never count a site they cannot see.
Claude-Session: https://claude.ai/code/session_01Y5ef34iUUMjmwnMpL9xgmB
Each site card carries an Artifacts badge (current / out of date / never
deployed) plus an "N of M instances out of date" line linking to Topology,
both read from the shared pending-updates snapshot. The snapshot loads after
the site list so a slow fleet evaluation never holds the cards, and a failure
degrades the badge to "unavailable" instead of taking the page down. Header
gains a Re-check button and an as-of caption; a successful Deploy Artifacts
forces a refresh so the badge flips immediately.
Claude-Session: https://claude.ai/code/session_01Y5ef34iUUMjmwnMpL9xgmB
Tallying filtered the whole deployed-instance set inside the site loop, making the
aggregate O(sites x instances) — the one per-entity loop in an otherwise batched
pass. Group once into a lookup and index per site. Also: ResolveArtifactState's
missing-hash branch is a defensive default, not the race the comment described
(the same site list feeds the hash computation), and the staleness batch omits an
instance undeployed mid-batch as well as one that never had a snapshot.
Claude-Session: https://claude.ai/code/session_01Y5ef34iUUMjmwnMpL9xgmB
Both repositories are scoped over one DbContext, so a failed stamp save left the
Site entries tracked as Modified and the deployment record's (uncaught)
SaveChangesAsync rethrew it — losing the audit row and failing a deploy in which
every site succeeded. Ordering rule now: audit row first, stamp after, inside its
own try/catch. Same in RetryForSiteAsync, where the site load is caught too so
nothing on the stamp path can reach the method's catch.
Also: the method-definitions canonicaliser matches the name property
case-insensitively (a naming-policy change at the producer would otherwise
silently turn the sort into a no-op), and ComputeCurrentArtifactHashesAsync
documents that a per-site query failure aborts the whole batch by design.
Claude-Session: https://claude.ai/code/session_01Y5ef34iUUMjmwnMpL9xgmB
One scoped read that answers "is anything waiting to be deployed?" for the whole
fleet: each site's stored artifact revision hash compared against what a deploy
would send right now (NeverDeployed / Current / OutOfDate), plus per-site
deployed/stale/unknown instance tallies from the batched staleness evaluation.
NeverDeployed is deliberately NOT a pending update — the operator has not deployed
yet — and exceptions propagate so the caller's single-flight cache can render
"unavailable" rather than invent an "up to date". Adds the validated
PendingUpdatesCacheTtl (60 s) that TTL will key off.
Claude-Session: https://claude.ai/code/session_01Y5ef34iUUMjmwnMpL9xgmB
ArtifactRevisionHashService (deterministic SHA-256 of the per-site
DeployArtifactsCommand, DeploymentId/timestamp excluded) stamps
Site.ArtifactRevisionHash/ArtifactsDeployedAt for every site whose deploy
succeeded; failed sites keep their previous stamp.
ComputeCurrentArtifactHashesAsync answers "what would it be now".
Claude-Session: https://claude.ai/code/session_01Y5ef34iUUMjmwnMpL9xgmB
DeploymentService.GetInstanceStalenessAsync shares a FlattenSession across
the batch, reads only the (InstanceId, RevisionHash) projection and pays the
snapshot read + I-1 List normalisation only for a mismatch, so the fleet
aggregate agrees exactly with the per-instance Topology badge. The I-1 logic
is extracted from GetDeploymentComparisonAsync unchanged.
Claude-Session: https://claude.ai/code/session_01Y5ef34iUUMjmwnMpL9xgmB
Site.ArtifactRevisionHash / ArtifactsDeployedAt (nullable, migration
AddSiteArtifactRevision) so the Sites page can tell whether a site's
system-wide artifacts are current. Not transported. Adds the
GetDeployedRevisionHashesAsync projection for the fleet-wide staleness batch.
Claude-Session: https://claude.ai/code/session_01Y5ef34iUUMjmwnMpL9xgmB
The Flow Debugger link sat as a lone top-level item after the Deployment
section because its policy (RequireFlowDebugger: Deployer + Administrator +
Viewer) is wider than the section's RequireDeployment (Deployer only), so it
did not expand/collapse with the group. The section is now gated on
RequireFlowDebugger — a superset of RequireDeployment — with the three
Deployer-only pages gated per item inside it, so every Deployer sees the
same section as before and a Viewer/Administrator without Deployer gets a
Deployment section holding only Flow Debugger.
Claude-Session: https://claude.ai/code/session_01Y5ef34iUUMjmwnMpL9xgmB
Joseph Doherty