Comment/XML-doc changes only, no code logic:
- strip project-tracking references (WP*, P*-*, row-NN, #NN, Task N) from
comments, keeping the explanatory rationale
- add missing summaries, <param>, <returns>, <typeparam> and <inheritdoc/>
- replace misused <inheritdoc/> with real docs; where <inheritdoc/> was
combined with useful remarks, move the rationale into // comments
- drop stale replicationActor <param> on DeploymentManagerActor ctor
Remaining 5 are false positives (CWE-1236 x2, failures-1, year-0001, M365).
Build 0 errors / 0 warnings.
Rig finding on the upgrade restart: both nodes' rows were older than the stale threshold
(the old build never refreshed them), so node-a's reporter pruned node-b's row while node-b
wrote a fresh one, and the replicated delete won. The reporter now prunes nothing during its
first refresh interval after start-up, and each mirror pass re-writes its own row if it no
longer holds what the mirror last wrote.
Each node now re-writes its mirror-status row every SecretMirrorStatusRefreshInterval
(10 min) as a heartbeat; SecretMirrorStatusReporter leaves off the health report and
deletes any row not refreshed for SecretMirrorStatusStaleAfter (1 h, validated >= 2x the
refresh). The delete is conditional on the checked_at it read, so a node that came back
keeps its fresh row; a returning node re-writes its row on its first pass.
Also: DeploymentDeadlineParityTests' stub replies with a real zero-failure summary (the artifact
aliases now derive their exit code from failureCount), and register row 127 reflects the merge
and the two new replicated site tables.
A DeployedSecretsMirror failure on the STANDBY node (e.g. a master-key mismatch
that stops a deployed secret decrypting there) was only logged on that node:
site health reports are sent by the active node only, so central never saw it.
- New 12th LocalDb-replicated table secret_mirror_status (node_name PK,
failures_json of {SecretName, Message}, checked_at; no FKs). Each node's
mirror writes only its own row, and only when its failure set changes plus
once per start-up (every write replicates via CDC). Best-effort: a failed
write is a Warning and is retried next pass. Node key = INodeIdentityProvider
NodeName, falling back to the machine name.
- SecretMirrorStatusReporter (SiteRuntime, every node, 30 s) reads every row
into ISiteHealthCollector.SetSecretMirrorStatuses -> additive nullable
SiteHealthReport.SecretMirrorStatuses (null = no data).
- Wire: SiteHealthReportDto.secret_mirror_statuses = 33 (presence wrapper so
null != empty); vendored CentralControl.cs regenerated; mapper both ways.
- Central UI Health card: 'Secret mirror failing' badge + compact per-node
node / secret / message list.
- Tests: mirror status write-on-change/first-pass/failure isolation, storage
round-trip, reporter/collector, DTO mapper round-trip incl. null/empty,
bUnit, replicated-table set (twelve) + FK guard wording.
Secret-store outage (final review M5): SiteSecretsCollector returns a store
read failure as SiteSecretsCollection.StoreError instead of throwing. A deploy
now fails only the sites that reference a secret (nothing sent, not stamped,
DeploySiteSecrets row with the error), and ComputeCurrentArtifactHashesAsync
omits only those sites (they read Out of date) instead of the exception
aborting the whole sweep and the fleet-wide pending-updates check.
Removal-only deploys: the site reply gains SecretsRemoved (proto field 7,
names only) - the deployed_secrets rows the deploy soft-removed. Central
carries it on SiteArtifactResult and writes the DeploySiteSecrets row with a
removed list whenever it is non-empty, so dropping a site's last secret:
reference is audited centrally rather than only in site logs.
site deploy-artifacts and deploy artifacts rendered the per-site summary and
always exited 0, so a script could not tell a failed site (unreachable,
missing or unreadable secret, partial apply) from a clean deploy. Both now
route through the failureCount rule deploy site already used.
I2: topology guide + Component-SiteRuntime + register row 127 state that a site pair using
deployed secrets must run LocalDb replication. I3: row 127 + topology guide state the upgrade
impact (whole artifact deploy fails and reads Out of date until referenced secrets exist
centrally). M6: audit row carries secrets/missing, no removed field; removal-only deploys write
no central row. M2/M3/I1 as-built notes; CLAUDE.md secrets bullet corrected.
Final review M3: SecretReferenceScanner.TryParseSecretReference (trimmed, case-insensitive
prefix, trimmed name, empty = not a reference, ${secret:} never matches) is used by the scanner
and by MxGatewayDataConnection's ApiKey resolution, so a padded or case-variant reference
resolves the name central ships. A bare 'secret:' ApiKey now fails closed. M7: unit test pins
stamped hash == ComputeCurrentArtifactHashesAsync with a non-empty Secrets list.
Final review M2: HandleDeployArtifacts now sends ApplyArtifactDataConnectionsToDcl after the
shipped secrets are written and mirrored (from a finally, so the push still goes out if the
secrets step fails), so a connection created by the same deploy resolves a newly shipped
secret on first connect. Definitions are still stored at the same point; response semantics
unchanged.
Final review I1: a name already ledgered at the shipped revision is now read back every
pass; a missing, tombstoned or edited local row is healed through the mirror-owned re-stamp
path (bump, apply, read back, invalidate) before verify, instead of a local delete being
misreported as a KEK mismatch that blocked every later deploy. Drift logs Warning once per
occurrence. M4: row_json that cannot be parsed becomes a Failed entry for that name via the
new SiteStorageService.GetDeployedSecretsTolerantAsync; an unreadable removed row is still
honoured by name. M10: invalidator ctor doc corrected.
The site applies DeployArtifactsCommand.Secrets as the full set (rows absent from it are
soft-removed and DeployedSecretsMirror deletes them on both nodes), while null means
'central shipped no secrets' and touches nothing. The command builder shipped null for a
site with no references, so dropping the LAST secret: reference never removed the secret
from either site node. Found by the Task 13 live gate (check 5: site-a-a and site-a-b
both still held sb-gate-mxgw r1, deployed_secrets.is_removed=0 after the removal deploy).
Central now always ships a list. Old sites ignore proto field 9; the artifact hash is
unchanged (empty and null hash identically); SecretsDeployed stays null and no
DeploySiteSecrets audit row is written when nothing shipped. Regression test
NoSecretReferences_ShipsEmptySecretsList_SoTheSiteRemovesPreviouslyDeployedOnes (red
before, green after). Spec notes in Component-DeploymentManager/SiteRuntime + the design
doc's as-built section.
Task 6 appended SiteSecretsCollector to ArtifactDeploymentService's constructor; the
NSubstitute class proxy in ManagementActorTests still passed the old ten arguments, so
the four DeployArtifacts_* tests failed with 'Could not find a constructor'. Caught by
the Task 13 full-solution run.
- SecretsRegistration: UsesCentralSharedStore predicate shared by AddScadaBridgeSecrets and the
new AddPreHostExpander, so a Central node's Layer-A ${secret:} expander composes against the
shared SQL store whenever Secrets:Store:Provider=SqlServer does — closing the gap where a
central migrated off the retired Grpc topology would boot-time-resolve config from per-node
SQLite while serving the shared store.
- Program.cs: expander block now calls AddPreHostExpander + resolves the store-agnostic
ISecretsStoreMigrator seam; stale Grpc-hub/replication comment rewritten.
- DeployedSecretsMirror.cs: stale "gRPC-hub follower half" doc comment rewritten to state the
now-permanent local-SQLite-only invariant.
- StartupValidator: Secrets:Replication:Enabled read as a raw string (any non-empty, non-"false"
value rejected) instead of GetValue<bool>, which threw a raw binder crash on "yes"/"1"; added a
CollectWarnings entry for leftover Secrets:Replication/Secrets:GrpcHub child keys once Enabled
is false/unset.
- Tests: +15 (AddPreHostExpander/UsesCentralSharedStore wiring, Replication:Enabled edge values,
leftover-key warnings). Full solution build and full Host.Tests green (593/593).
Fix round 2 (review task-9-rereview-1.md), both Minors:
1. Clock skew. ApplyAndConfirmAsync now decides the tombstone-vs-out-of-band split by LEDGER
OWNERSHIP (any mirrored_secrets entry for the name, at any revision), not merely by whether
the local row happens to be a tombstone. A live local row this node already owns but whose
ciphertext no longer matches the newly shipped revision (e.g. an earlier tombstone-recovery
re-stamp landed ahead of a since-shipped central UpdatedUtc, under site/central clock skew)
is now re-stamped and re-applied exactly like a tombstone, instead of being misreported as
someone else's "newer local value" and left permanently un-converged. The out-of-band Failed
message is now only ever produced for a name that is NOT in the ledger. New test:
Reconcile_ClockSkewedRecoveredRow_StillConvergesOnNextCentralRevision (a skewed future
re-stamp followed by a genuinely new central revision converges and re-invalidates the
cache).
2. Real-cipher integration test. Added DeployedSecretsMirrorRealStackTests, running the mirror
against the REAL ZB.MOM.WW.Secrets stack (SqliteSecretStore + AesGcmEnvelopeCipher +
DefaultSecretResolver over a real temp-file SQLite database, plus a hand-written
IMasterKeyProvider test double mirroring the library's own) instead of in-memory fakes:
encrypt -> SecretArtifact -> apply -> delete (tombstone) -> re-add the same artifact -> the
recovered row resolves to the original plaintext. Pins the "UpdatedUtc/Revision are not
AEAD-bound" property against the real cipher rather than by source inspection alone.
Test-only package reference: ZB.MOM.WW.Secrets (SiteRuntime itself still depends only on
ZB.MOM.WW.Secrets.Abstractions).
Verify: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter FullyQualifiedName~DeployedSecretsMirrorTests (25/25); full SiteRuntime.Tests suite 878/878.
Fix round 1 (review task-9-review.md), all rulings:
1. (Important 1) ApplyReplicatedAsync is no longer trusted blindly. Every apply is
confirmed by a store.GetAsync read-back before ledgering: a tombstoned local row
(routine remove-then-re-add) is safely re-stamped past the tombstone and re-applied
(confirmed neither UpdatedUtc nor Revision is AEAD-bound in AesGcmEnvelopeCipher, so
the bump cannot break decryption); a live local row with different ciphertext (a
strictly-newer out-of-band secret) is left untouched, never ledgered, never verified
against, and reported with a distinct Failed message so a later central tombstone can
never delete it via this mirror.
2. (Important 2) Per-row try/catch (rethrow only on ct.IsCancellationRequested) so one
poisoned row (e.g. an unrecognised ContentType from a newer central) no longer blocks
every row after it.
3. (Important 3) Optional ISecretCacheInvalidator, invalidated after every successful
apply/delete and before verify, so a rotated/removed secret's plaintext cannot be
served stale for the remainder of the resolver's TTL.
4. (Minor 4) Loop and VerifyAsync OperationCanceledException handling now filters on
ct/stoppingToken.IsCancellationRequested specifically, rather than swallowing every OCE.
5. (Minor 5) Class remarks note the mirror assumes a node-local ISecretStore; Task 11
will reject SqlServer secrets replication on the Site role.
6. (Minor 6/7) Added cancellation + concurrent-callers tests, dropped the redundant
FailFor (tests now use ThrowFor/ReturnNullFor to exercise distinct decrypt-failure
paths), and Dispose() now disposes the gate SemaphoreSlim.
Verify: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter FullyQualifiedName~DeployedSecretsMirrorTests (24/24); full SiteRuntime.Tests suite 876/876.
ArtifactDeploymentService now resolves each site's secret: references (via
the new SiteSecretsCollector, wrapping ISecretStore) into DeployArtifactsCommand.Secrets,
never plaintext. A referenced secret that is absent or tombstoned fails
that site's deploy before anything is sent, with a named error, while the
rest of the batch still deploys; ComputeCurrentArtifactHashesAsync instead
folds a MissingSecrets marker into the hash so such a site reads Out of
date. Central logs what it ships (name@revision) and errors on what's
missing, never material.
Also updates the two other direct ArtifactDeploymentService construction
sites (PendingUpdatesEvaluatorTests, CentralUI SitesPageTests) for the new
constructor parameter.
Second vd03 window of the day: no migrations, appsettings.Site.json gains ScriptFileAccess:AllowedRoots
(pushed over SMB before the swap), sample recipes in E:\ApiInstall\ScadaBridge\recipes. The Delmia
test receiver now reads + parses the downloaded file; real HTTP R100 -> 12 values loaded, R900 -> rejected
with line numbers. verify-mes-test-machine.sh: D1's cleared check reads through BTDB, so it only runs
with group C (the rig runs VERIFY_GROUPS=AD).
Script file access (design docs/plans/2026-09-28-script-file-access-design.md): Files.ReadTextAsync /
ReadLinesAsync / ExistsAsync on site ScriptGlobals (+ Instance.Files), backed by the site-singleton
ScriptFileReader. The one host-mediated exception to the System.IO ban - scripts still never name a
System.IO type. OFF by default: ScadaBridge:SiteRuntime:ScriptFileAccess:AllowedRoots (empty = every call
throws Disabled) + MaxFileBytes (1 MiB, validated 1..64 MiB, roots must be absolute). Guard: fully
qualified, no device paths / ADS, canonical path strictly inside a root (directory-boundary match), no
reparse point on the file or any directory below the root, size cap before and during the read.
Errors are one Commons type, ScriptFileAccessException(+ScriptFileAccessFailure); Commons.Types.Scripts
joins the default script imports on every surface so it binds by short name. Mirrored on
ScriptCompileSurface (CompileFiles, parity-pinned) and the Test Run sandbox (binds, refuses Disabled).
Inbound (central) scripts do not get it.
Rig: docker/recipes (sample RecipeValueName,RecipeValue files incl. an invalid one) bind-mounted
read-only at /app/recipes on both site-a nodes with AllowedRoots [/app/recipes]. The TestMesMachine
SimDelmiaRecipe simulator now reads + parses the downloaded file, converts each value to its recipe tag's
type and loads it all-or-nothing (14 new TestDelmiaReceiver attributes); verify-mes-test-machine.sh
group A covers accept / re-download / invalid lines / NotFound / OutsideAllowedRoots, VERIFY_GROUPS
selects groups (rig: AD).
vd03 (DEV) upgraded 2b9a7c3c -> fe89363b via the deployment-only winrm channel: COPY_ONLY backup,
55->57 delta script (AddAlarmMesReportable, AddSiteArtifactRevision), ~53 s window, snapshots
identical to baseline, artifacts deployed. Register row 126 now covers the QA pair only.
docker/seed-mes-test-machine.sh (idempotent) builds a standalone TestMesMachine base template -
Left/Right TestMESReceiver + TestDelmiaReceiver (prod attribute names, STATIC), computed alarms
with IsMesReportable, CvdReactor-compatible MesMoveIn/MesMoveOut/ProcessRecipeDownload/AlarmStatus
scripts ported to SetAsync+WaitAsync, and Galaxy-side simulators with reject/no-response knobs -
plus instances SBTEST01/02 and the SimpleAlarmStatusRequest (was a stub) / AlarmStatusRequest
inbound methods in the legacy /mes/(simple)alarmstatus shapes, resolving machines through BTDB.
BTDB rows SBTEST01=990001 / SBTEST02=990002 added on vd03.
docker/verify-mes-test-machine.sh drives the real inbound methods via flow-debug: PASS 25/25.
The 2026-09-11 hold (WinRM exec tripped the wonder boxes' security software) is lifted for
deploys only: preflight, backup, stop/swap/start, migration, post-deploy verify, rollback.
Ad-hoc diagnostics and deployed-version checks use SMB/HTTP/SQL instead. psbridge stays retired.
Checked against the legacy mesapi source (MesNotifier/MesServices/Tag) plus ServiceStack/OrmLite
6.0.2 in a scratch harness:
- unhandled exceptions are an HTTP 500 ServiceStack ErrorResponse, not WasSuccessful=false
- no matching alarms = success with an empty list; failures don't short-circuit
- AckDT is null only for a null MXAccess value, not 'when unacked'
- tag type-conversion failures are silent (InAlarm false -> alarm drops out)
- simplealarmstatus SAPID edge cases: null never matches, "" matches, duplicates -> arbitrary TOP 1
- 'test' API keys hit the production BatchDB (TestDb routing never reaches MesNotifier)
- latent SetResult-after-timeout race in the MXAccess data-change callback
OnParametersSetAsync awaits the principal. On a real cookie session that await
suspends, so the FIRST render ran with _visible still false and
OnAfterRenderAsync(firstRender: true) skipped EnsurePollingStarted(); the
render that followed the resolve was firstRender: false, and nothing else
started the poll unless AuthenticationStateChanged happened to fire. The block
therefore rendered and sat on 'Checking for updates…' for the life of the
circuit. OnParametersSetAsync now starts polling itself when visible (the
start is idempotent and the OnAfterRender call is kept).
Also corrects the comments that called the non-forced
GetAuthenticationStateAsync cheap: on CookieAuthenticationStateProvider both
entry points revalidate against the database and differ only in failClosed, so
the saving is not asking the provider on a tick and reusing the cascaded state
the router already resolved. The provider fallback is a real refresh, is only
taken when no cascade is present (AddCascadingAuthenticationState is
registered, so it is dead in production), and is now resolved at most once per
auth state rather than on every parameter set.
Claude-Session: https://claude.ai/code/session_01Y5ef34iUUMjmwnMpL9xgmB
The gate could pass vacuously. Every site-list read went through a bare
`ROWS=$(site_rows)` with neither the exit status nor the row count checked, and
an empty result satisfied every downstream assertion for free: no rows means no
malformed hash for B1/B2 to report, and a diff of two empty strings is no
difference at all for C1/D2. A down management API would therefore have read as
a clean gate.
Reads now go through read_rows, which fails loudly on a non-zero CLI exit
(site_rows exits 2 on an empty list, and pipefail surfaces a failed call) and
aborts unless the line count equals the site count established in step A —
keeping the running tally so the abort still prints PASS/FAIL. Verified against
an unreachable URL: FAIL 1, exit 1.
The join(1)/cut/diff comparisons assumed lexicographic id order matches numeric
order, which breaks once a site id reaches 10 alongside single-digit ids (and BSD
join silently drops the unmatched rows rather than complaining). Replaced with a
python3 comparison keyed by integer id that also asserts the id SETS match, so a
site appearing or vanishing between two reads invalidates the comparison instead
of quietly shrinking it. Same for the summary table.
Also: set -euo pipefail like every sibling script (the cmd && pass || fail guards
keep -e safe, and -e does not cover var=$(cmd) — hence the explicit checks
above); the cleanup trap is armed BEFORE the shared-script create and falls back
to finding the probe by its unique name, so a create that succeeds server-side
but whose id we fail to parse can no longer orphan a probe; mktemp instead of
/tmp/pu-deploy.$$.
PendingUpdatesTests: the rail-scroll check asserted scrollHeight >= clientHeight,
which holds for every element. It now asserts what the pane is actually for —
the pane is bounded by the viewport and the rail footer holding the indicator
stays on screen, so a nav that grows the rail instead of scrolling fails. Live
values are genuinely overflowing (scrollHeight 1196 vs clientHeight 680, footer
bottom 765 of 900), so the assertion has real content.
Re-run against the rig: PASS 13 / FAIL 0; Playwright 3/3.
Claude-Session: https://claude.ai/code/session_01Y5ef34iUUMjmwnMpL9xgmB
SiteScopeService's instance methods resolve the AUTHORITATIVE state — a
fail-closed, DB-backed group-to-role re-mapping. The indicator called them on
every 30s tick, and it lives in MainLayout, so that was one or two forced
session revalidations per open circuit per page, forever: exactly the
per-circuit cost IPendingUpdatesCache exists to avoid.
Extracts the pure rule as SiteScopeService.Resolve(ClaimsPrincipal) (no I/O;
ResolveAsync now delegates to it, no behaviour change for existing callers).
The indicator takes the cascaded AuthenticationState — falling back to a
NON-forced GetAuthenticationStateAsync, as Templates/FlowDebugger do — and
computes roles plus scope once per auth state, re-running on
AuthenticationStateChanged so a mid-session revocation hides the block
instead of leaving it stale for the life of the circuit. Also routes the
first load through PollGate, adds aria-hidden to the icons, and comments the
off-renderer field writes and the InvalidOperationException catch.
Claude-Session: https://claude.ai/code/session_01Y5ef34iUUMjmwnMpL9xgmB
Rig rebuilt from this branch (image built at d8979664; migration
20260914173147_AddSiteArtifactRevision applied by central-b on boot).
verify-pending-updates.sh drives the CLI end to end: the site list carries the
artifact-revision fields, a fleet-wide deploy stamps every site with a
sha256:<64 hex> hash and a fresh timestamp, creating a shared script leaves the
stamp alone until the next deploy and then moves EVERY site's hash, and deleting
it again returns every hash byte-for-byte to its pre-create value — so the hash
is content-addressed, not deploy-addressed. PASS 13 / FAIL 0.
Step A asserts the SHAPE rather than key presence: the management API serializer
omits null properties, so on a never-deployed site the two keys are absent, not
present-and-null. Step E is a soft reachability check only; the badge text and
the rail indicator land after the Blazor circuit's post-render load.
PendingUpdatesTests covers that browser half — the rail footer settles off
"Checking for updates…", the Sites page carries an Artifacts badge per site card
plus a working Re-check control, and the rail nav's scroll pane computes
overflow-y: auto at lg+. 3/3 passed against the rig (rail read "Up to date",
3 badges over 3 cards, all "Artifacts current").
Claude-Session: https://claude.ai/code/session_01Y5ef34iUUMjmwnMpL9xgmB
Joseph Doherty