Record of the 2026-11-12 config-only change on wonder-app-vd03: MxGateway__ApiKeyPepper
added to the MxAccessGw AppEnvironmentExtra block (same value, sha256 prefix match),
removed from Server\appsettings.json; authenticated round-trip with a pre-change key,
ScadaBridge MxGateway link unchanged. Runbook and configuration doc now say where the
pepper lives on each wonder box.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
New optional MxGateway:Ldap:ServerCertificateThumbprintSha256 (hex SHA-256
of the DER cert; case/colon/dash/space-insensitive; comma/semicolon-separated
string or array, so a rotation can carry old + new). When set, the dashboard's
shared LDAP provider gets a RemoteCertificateValidationCallback via
LdapOptions.ServerCertificateValidationCallback (Auth 0.4.1) that accepts the
server certificate iff its thumbprint matches a pin, ignoring chain/name
errors only then. Unset keeps default OS trust-store validation untouched.
The callback is an init-only, code-only property, so the ILdapAuthService
registration from AddZbLdapAuth is replaced with one built from the bound
options plus the callback. Pins are read straight from configuration because
the binder silently binds the scalar form onto a list as empty.
GatewayOptionsValidator rejects a pin that is not 64 hex chars after
normalization and any pin with Transport=None; the Production hard stops are
unchanged. Pins are shown on the Settings page. Docs: GatewayConfiguration.md,
glauth.md.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
- New MxGateway:Worker:WorkerHeartbeatGraceSeconds (default 15), exported as
MXGATEWAY_WORKER_HEARTBEAT_GRACE_MS and read by WorkerPipeSessionOptions.FromEnvironment
(a grace the ceiling does not exceed falls back to the default so the worker
still starts). Validation now checks WorkerHeartbeatStuckCeilingSeconds against
the configured grace; the copied WorkerWatchdogGraceSeconds constant is gone.
- EffectiveWorkerConfiguration and the dashboard Settings page now show the worker
heartbeat send interval, watchdog grace and stuck ceiling (render test added).
- Docs updated.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The worker's heartbeat send interval (5 s) and its own STA-watchdog stuck
ceiling (75 s) were compile-time defaults the gateway never reached, and
nothing checked that the gateway's HeartbeatGraceSeconds exceeded the real send
interval, so a misconfigured grace could kill healthy workers.
- New MxGateway:Worker options WorkerHeartbeatSendIntervalSeconds (default 5)
and WorkerHeartbeatStuckCeilingSeconds (default 75), exported by
WorkerProcessLauncher as MXGATEWAY_WORKER_HEARTBEAT_SEND_INTERVAL_MS and
MXGATEWAY_WORKER_HEARTBEAT_STUCK_CEILING_MS, the same env-var path the other
worker settings use.
- The worker's production WorkerPipeSession reads them through
WorkerPipeSessionOptions.FromEnvironment; a missing/invalid value, or a
ceiling not above the worker's 15 s watchdog grace, keeps the default rather
than failing the worker.
- Startup validation: HeartbeatGraceSeconds > WorkerHeartbeatSendIntervalSeconds;
send interval > 0; worker ceiling > 15 s; both at most 86400 s.
- Docs: GatewayConfiguration, WorkerProcessLauncher (env table now also lists
the previously missing MXGATEWAY_ALARM_ACK_CLIENT_NAME), worker design doc.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
R21-K12: WorkerFrameWriter.WriteBatchAsync allocated a TaskCompletionSource
(plus its Task) per frame on the event drain hot path. PendingFrame now holds
its terminal state itself (Interlocked pending/succeeded/faulted/canceled) and
only single-frame WriteAsync frames, which race their completion against the
write lock, get a TCS. A batch caller reads each frame's outcome after its own
drain pass: every frame a pass claims is terminal before that pass releases the
lock, so nothing is left to await. First-failure-in-batch-order, tombstoning,
claim/cancel interlock and flush coalescing are unchanged.
R21-K13: WnWrapAlarmConsumer.ParseSnapshotXml streams the GetXmlCurrentAlarms2
payload with an XmlTextReader (same defaults XmlDocument.LoadXml uses:
Normalization off, DTDs parsed, no resolver; Significant whitespace only)
instead of building a full XmlDocument every 500 ms poll.
Behaviour is pinned by new tests: batch first-failure ordering (own drain and
drained by another lock holder), batch cancellation tombstoning, batch flush
failure, and a differential test running the replaced XmlDocument parser as an
oracle over a 48-payload corpus (CRLF, CDATA, entities, DTD, namespaces,
nesting, malformed). Mutation checks on windev: 8 of 8 injected regressions
(whitespace, normalization, DTD, namespace, CDATA, batch last-failure, batch
ignored failures, ignored tombstone) fail the new tests.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The replay ring was bounded only by ReplayBufferCapacity (1024 events) and
ReplayRetentionSeconds. Event size is unbounded up to the worker frame size,
so a burst of large-array events could pin gigabytes per session while the
count looked small.
New MxGateway:Events:ReplayBufferMaxBytes (default 64 MiB, 0 disables) caps
the summed serialized size (MxEvent.CalculateSize, measured only when the cap
is on). Over the cap the oldest entries are evicted, exactly like the count
cap, so a resuming subscriber sees the existing ReplayGap path; an event bigger
than the whole cap is not retained. Every eviction (count, age or byte) now
clears its slot, so an evicted event is released immediately instead of
staying referenced until the slot is reused.
The 64 MiB default is a judgement call: it never binds for scalar traffic
(1024 x a few hundred bytes) and only trims large-array bursts.
Tests: SessionEventDistributorTests byte-cap eviction, single oversized event,
cap 0 = count only (the first two fail with the cap enforcement removed);
validator rejects a negative value. Docs: GatewayConfiguration.md, Sessions.md,
DesignDecisions.md; appsettings.json carries the key.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
R21-O14: MxGateway:Worker:HeartbeatIntervalSeconds was documented as the
worker heartbeat send interval, but it never reached the worker (which sends
on its own fixed 5 s) and only set the gateway's HeartbeatCheckInterval. It is
renamed HeartbeatCheckIntervalSeconds (default 5, unchanged). The old key stays
as a deprecated alias with the same meaning so existing appsettings keep
working; setting both is a validation error. The grace-vs-interval check now
compares against the resolved check interval.
R21-O15: WorkerClientOptions.EventChannelFullModeTimeout and
HeartbeatStuckCeiling were always their compile-time defaults. They are now
MxGateway:Worker:EventChannelFullModeTimeoutMilliseconds (default 5000, > 0)
and HeartbeatStuckCeilingSeconds (default 75, >= HeartbeatGraceSeconds). The
factory's mapping moved into SessionWorkerClientFactory.CreateWorkerClientOptions
so every field is set from configuration and is testable. The effective-config
record shows the resolved check interval and both new values.
Docs: GatewayConfiguration.md, GatewayProcessDesign.md sample,
MxAccessWorkerInstanceDesign.md heartbeat section; appsettings.json uses the
new keys.
Tests: CreateWorkerClientOptions mapping + defaults (fail under a mutated
mapping), alias honoured, both-keys conflict, and bounds for the new options.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Add Dashboard DisableLogin/AutoLoginUser/CookieName, Alarms:Fallback and Tls,
matching the option classes and the per-option tables below the block.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Set retainedFileCountLimit=30, fileSizeLimitBytes=50 MiB and rollOnFileSizeLimit=true on the
file sink in appsettings.json and appsettings.Development.json; document under a new Logging
section in docs/GatewayConfiguration.md; add a guard test over the shipped config.
WorkerProcessStartedProbe only checks HasExited, so a retry could never change
the outcome. Remove StartupProbeRetryAttempts/StartupProbeRetryDelayMilliseconds,
their validation, the launcher's Polly retry pipeline and the docs; the probe now
runs once under StartupTimeoutSeconds. Stale config keys are ignored by the binder.
NT AUTHORITY\NETWORK marks a network logon, not a remote caller. On windev
a gateway run from an SSH session carries it in its own token, so the deny
entry locked the gateway's same-machine workers out (UnauthorizedAccess on
connect; 15 fake-worker tests failed). The ACL now holds exactly one entry:
FullControl for the gateway's user.
On Windows each worker pipe is created via NamedPipeServerStreamAcl with a
protected DACL granting FullControl only to the gateway process user (the
worker is launched under the same account) and denying NETWORK logons. On
non-Windows hosts (test matrix) the pipe uses PipeOptions.CurrentUserOnly.
Tests assert the factory ACL and the ACL read back from the created pipe
(Windows), and that a same-user client still connects.
- Sessions.md / GatewayConfiguration.md: the dashboard Browse live-data session is
unary-only, is reaped by UnattachedSessionGraceSeconds when idle, and reopens via
EnsureReadyAsync; only the alarm monitor is exempt.
- GatewayConfiguration.md: a refused second subscriber gets ResourceExhausted, not AlreadyExists.
- Grpc.md: rewrite the event streaming model around the SessionEventDistributor pump and
replay ring; add EventSubscriberLimitReached to the mapping snippet; describe what
clients see while the gateway is stopping.
- The metrics scheme skips its dashboard-cookie fallback while
Dashboard:DisableLogin is on, so the dev auto-login handler cannot
satisfy RequireAuthentication=true.
- Invoke logs client_correlation_id <-> worker envelope id once at Debug.
- MetricsEndpointAuthenticationTests drives the real pipeline: anonymous
loopback 401 (no /login redirect), Viewer cookie 200, admin key 200,
non-admin key 403, DisableLogin 401, /health/active 200.
- Docs: DisableLogin behaviour; the /metrics limiter partition is per IP
vs gRPC per connection, so only the per-key aggregate is shared.
- Open path: the attach gate now waits on CancellationToken.None and the
manager discards (kills + disposes) the late worker client on ANY
exception out of the attach step, including the caller's own cancellation
(checked first, so an abandoned open never yields a session). Previously
an OperationCanceledException from the gate wait escaped before the
discard and orphaned the x86 worker. The gate Release is guarded against
ObjectDisposedException.
- Unattached grace: AttachInternalEventSubscriber sets a separate latch so
the gateway's own alarm-monitor session (internal subscriber only) is
never reaped; the idle clock starts at Ready (MarkReady stamps
_readyAtUtc) instead of at open, so a slow worker start does not eat the
window.
- KillWorkerAsync / CloseSessionCoreAsync: an ObjectDisposedException from
the close gate on a session that is no longer registered means a
concurrent close already removed, accounted and disposed it; both now
return AlreadyClosed with no fault, no second gauge decrement and no sweep
failure. The general kill catch only decrements the gauge when the
session is still registered. The sweep no longer counts AlreadyClosed
results among the sessions it closed.
New MxGateway:Metrics:RequireAuthentication (default false). When true,
/metrics requires an API key with the admin scope (verified through the
cached verifier and the shared API-key failure limiter) or a dashboard
login with a real dashboard role; anonymous localhost and
Authentication:Mode=Disabled do not qualify. An anonymous scrape gets a
plain 401 Bearer challenge, a throttled peer 429. /health stays anonymous
because the off-repo Overview dashboard polls /health/active.
MapZbMetrics returns IEndpointConventionBuilder, so the policy is attached
with RequireAuthorization.
A client that opens a session and vanishes before it ever streams (e.g.
OtOpcUa's GalaxyDriver on a failed start) pinned a session slot and a live
x86 worker for the full DefaultLeaseSeconds (1800 s), because detach-grace
only applies to sessions that once had a subscriber.
MxGateway:Sessions:UnattachedSessionGraceSeconds (int, default 0 = off,
validated >= 0) makes the existing lease sweep reap a Ready session that
has never attached an event subscriber once it has been idle (no unary
activity) for the window, with the close reason unattached-grace-expired.
Ready-only, idle-from-last-activity and first-subscriber-cancels keep it
from catching slow starts or unary-only clients.
GatewaySession.ExtendLease had no production callers and did not fit the
rule, so it is removed; tests move leases through TouchClientActivity.
The 2026-10-18 live test on the dev rig showed the opt-in managed ack path
cannot work as built: AlarmClient.RegisterConsumer returns -65 alongside the
worker's wnwrap reader (the worker then silently falls back to wnwrap), the
managed client does not recognise the wnwrap GUID even standalone, and the
gateway's GUID arm sends empty identity fields. Replace the docs' claim that
the path works with the measured results and tell operators to leave
MXGATEWAY_USE_MANAGED_ALARM_ACK unset.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review fixes for the opt-in managed GUID ack:
- Subtag-mode GUID acks (forced subtag, or failover on the standby) stay on
the consumer; their GUIDs are synthetic and only the subtag consumer can
resolve them. Forced-subtag sessions no longer build a managed client.
- A failed managed Initialize/Register/Subscribe drops back to the wnwrap
ack path instead of failing the alarm subscription.
- Drop stale crefs to a nonexistent Unverified member in
WnalNativeAckConsumer.
- Unit tests for routing, fallback, and disposal; document
MXGATEWAY_USE_MANAGED_ALARM_ACK.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The subtag fallback's coverage was fixed at SubscribeAlarms time and never
re-examined. AlarmWatchListResolver deliberately swallows a Galaxy Repository
outage and returns whatever configuration contributed, so a discovery failure at
startup left the monitor subscribed with an empty watch-list for the rest of its
life. In Auto mode the alarmmgr primary keeps working, so nothing looks wrong
until the failover that needs the standby finds it covering zero alarms — with a
single LogWarning emitted possibly weeks earlier as the only evidence.
ResolveAsync now returns an AlarmWatchListResolution (targets + whether discovery
was attempted and whether it failed, with the failure message), which is what the
caller needed to tell "discovery produced nothing" from "nothing was configured".
GatewayAlarmMonitor grades that resolution against the effective mode and folds
the verdict into AlarmProviderStatus, so it reaches the feed, the dashboard badge
and every late-joining subscriber:
- zero targets while the mode can need them (Auto/ForceSubtag) -> degraded with
SubtagWatchListEmpty; discovery enabled but failing adds SubtagDiscoveryFailed
and the message. ForceAlarmManager builds no subtag consumer and is not graded.
- degraded is now the OR of "subtag is serving" and "the watch-list covers
nothing", so the Auto case where alarmmgr is healthy and the standby is empty
is visible. DashboardAlarmProviderStatus renders that combination as the amber
"Alarm Manager (no subtag standby)" badge rather than the failover label, which
would have named the wrong live provider.
- on the failover edge (ApplyProviderModeChangeAsync, toMode == Subtag) the
monitor re-resolves BEFORE publishing the status and reports any difference as
SubtagWatchListStale with both counts.
The re-resolution is deliberately NOT re-sent. The worker rejects a second
SubscribeAlarms while subscribed (AlarmCommandHandler.Subscribe), and the only
expressible replacement — UnsubscribeAlarms then SubscribeAlarms — rebuilds
FailoverAlarmConsumer, which restarts on the primary that just failed and
discards the warm standby state, so it would flap the feed rather than repair it.
The finding's other alternative (treat empty coverage as a lifecycle failure) was
rejected too: it would crash-loop the monitor every 5 s on a persistent GR outage
and take the working alarmmgr feed down with it. A monitor restart re-resolves and
re-sends, and the status now says so.
Docs: docs/AlarmClientDiscovery.md gains the grading rules and corrects two stale
claims (the resolver method name, and "re-runs discovery on its reconcile cadence
and pushes an updated watch-list", which never happened); gateway.md gains the
standby-coverage visibility paragraph; docs/GatewayConfiguration.md records the
consequence on the UseGalaxyRepository and Mode rows.
Tests: SubtagCoverage_WhenDiscoveryFailsAtSubscribe_ProviderStatusIsDegradedWith-
Reason and SubtagCoverage_WhenDiscoveryRecoversByFailover_ReResolvesAndReportsDrift
drive the fake-worker monitor; two resolver cases pin the new provenance fields.
The pre-existing dashboard case for degraded+ALARMMGR now asserts the new label.
The worker budgeted a DrainEvents reply against the negotiated pipe frame
maximum less one 64 KiB reserve, but the gateway returns that same
MxCommandReply verbatim through Invoke, where MaxSendMessageSize applies. The
two limits were coupled only from below (pipe >= gRPC + reserve), so any
configuration the validator permitted with a wider gap let the worker pack a
reply the pipe carried and the gRPC layer then refused with ResourceExhausted --
after Drain had already removed the events from the worker queue, losing them to
both the caller and the event stream.
Option (a) from the finding: close the coupling at both ends.
- GatewayOptionsValidator adds the upper half of the headroom rule
(Worker:MaxMessageBytes <= Protocol:MaxGrpcMessageBytes + 2 x reserve), with a
message naming the DrainEvents budget as the reason and the two knobs to move.
- WorkerPipeSession.ResolveDrainReplyByteBudget subtracts the reserve twice --
once for the pipe frame, once so the MxCommandReply wrapper clears the gRPC
send cap -- keeping the MaxMessageBytes/2 floor that makes small frame maxima
usable.
The two changes are load-bearing for each other: the validator band is what
makes "frame max - 2 x reserve" provably at or below the gRPC cap, and the
doubled subtraction is what makes the band sufficient.
Docs updated in the same change (GatewayConfiguration.md both option rows plus
the headroom prose, WorkerFrameProtocol.md drain budget, gateway.md DrainEvents
byte axis). Tests: new GatewayOptionsValidatorTests cases for the upper bound
(reject) and both edges of the band (accept); GatewayOptionsTests' override case
now moves both message-size knobs together, since the band makes moving one
alone illegal.
The worker half cannot be built on macOS; verified on windev separately.
Committed together because the two changes share the same doc paragraphs (the ACL
section of GatewayDashboardDesign.md, the GroupToTag / AllowAnonymousLocalhost rows
of GatewayConfiguration.md, and CLAUDE.md's authentication paragraph), and splitting
them would leave one of the two describing behaviour that does not yet exist.
SEC-41 — tag grants no longer trusted from login-stamped claims
---------------------------------------------------------------
DashboardSessionAcl read the zb:dashboardtag claims stamped into the cookie at
login. GatewayOptions binds once at startup, so applying a GroupToTag edit needs a
restart — but the restart does not clear the auth cookie (data-protection keys
persist), so a browser returning afterwards re-authenticated with tag claims
resolved under the OLD map. A revoked grant therefore survived on the cookie seam
for the sliding 8-hour idle life, renewable indefinitely by activity, while the
hub-token seam had no such gap because its mint re-resolves.
CanViewSession now re-resolves the grant on every subscribe from the caller's
mxgateway:ldap_group claims through DashboardGroupTagMapping.MapGroupsToTags
against the currently bound map — the same one call HubTokenService.Issue makes.
Fallback (documented on the class and in the design doc): a principal carrying NO
group claims falls back to its stamped tag claims. That arm exists for the
hub-token principal — HubTokenService.Validate rehydrates roles and tags but not
groups, so re-resolution already happened at mint, one lifetime ago at worst. It
cannot re-open the cookie hole: DashboardAuthenticator stamps group and tag claims
from one group list, so a cookie principal holding a tag claim always holds the
groups it came from, and a login yielding no groups yields no tags. A principal
with neither kind of claim stays an empty-grant Viewer.
SEC-40 — the alarm feed requires a real authenticated dashboard role
--------------------------------------------------------------------
AlarmsHub sat behind HubClientsPolicy, which the read-only environment bypasses
satisfy, so an anonymous-localhost caller or a Mode=Disabled deployment received
galaxy-wide alarm metadata — alarm reference (a full tag path), state, severity,
category, area, acknowledging operator — for the whole subscription expression.
Decided: tighten the gate, keep the scope. New AuthenticatedDashboardRoleRequirement
+ its handler, and a MxGateway.Dashboard.Alarms policy built from it. Because
DashboardAuthorizationHandler is typed to DashboardAuthorizationRequirement, it is
never invoked for the new type, so neither Authentication:Mode=Disabled nor
anonymous localhost can satisfy the policy — the same structural device the mounted
Secrets UI already relies on. The policy keeps both schemes so external hub clients
can still authenticate with a bearer token, and DisableLogin auto-login is
unaffected (it mints a genuinely authenticated both-roles principal).
The /alarms page carries the same policy, not just the hub: it reads the same
central monitor in process, so gating only the hub would have left the disclosure
one navigation away. The side rail hides the Alarms link for callers the policy
denies, matching the Secrets link precedent. AlarmsHubPublisher is unchanged and
stays unfiltered by design — audience is decided at hub admission, and the monitor
carries no per-session provenance to narrow by.
Alarm metadata deliberately remains galaxy-wide for authenticated Viewers: an
AlarmFeedMessage has no session id or owning key for the SEC-25 tag intersection to
work against, and equating a visibility tag with an alarm-reference prefix would
turn GroupToTag into the data-access constraint it explicitly is not.
Tests
-----
DashboardSessionAclTests: the SEC-41 regression — a cookie-style principal whose
stale tag claim is no longer granted by any of its groups under the current map is
denied; plus the allowing direction (including with no stamped tag at all, which
proves the decision comes from the groups), the ignore-case corner, group claims
that map to nothing, and the hub-token fallback asserted against an EMPTY map so
dropping the fallback would fail loudly.
AlarmFeedAuthorizationTests (new): both handlers run together in the most permissive
environment (Mode=Disabled + loopback + AllowAnonymousLocalhost) do not satisfy the
alarm requirement, with the bypassable requirement as the control so the negative
cannot pass because the bypass was misconfigured; authenticated Viewer/Admin do
satisfy it; an unauthenticated role claim does not; the registered policy carries
the non-bypassable requirement and both schemes; and both readers — the hub type's
[Authorize] and the /alarms endpoint metadata — carry it.
SecretsNavRenderTests: registers the new policy (an unregistered policy name throws
rather than denying in an AuthorizeView) and gains the Alarms-link negative/control
pair.
Verified: dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx (zero warnings);
filtered run of the touched classes — 92 passed, 0 failed.
Follow-up to the Server-040 comment restoration (d3ac527). The restored
comment in DashboardGroupRoleMapping records that the shared
ZB.MOM.WW.Auth.Ldap provider hands the mapper groups already stripped to
short RDN names, so a full-DN GroupToRole *key* can never match. Three
doc sites still told operators the opposite, and the tag mapper carried
the same consequence with no comment at all.
- docs/GatewayConfiguration.md: GroupToRole and GroupToTag rows both said
keys may be 'short CN or full DN — leading-RDN match'. Corrected to
short group names only, with the reason (pre-stripped groups) and a
note that the mapper's full-string-then-RDN fallback only matters to
non-library callers of the IGroupRoleMapper<string> seam.
- docs/GatewayDashboardDesign.md: the login-flow bullet said the user's
memberOf '(or short CN)' is matched against GroupToRole; it now
describes what actually happens — groups arrive short and are matched
against short-name keys.
- DashboardGroupTagMapping: short comment at the lookup pointing at the
DashboardGroupRoleMapping comment, since it reuses ExtractFirstRdnValue
and inherits the same full-DN-key-unsupported consequence.
Both doc rows are pre-existing on main — they predate this branch and
were not introduced by the Server-040 work; only their incorrectness was
surfaced by it. Comment/doc-only; no logic changes.
The per-session dashboard event ACL shipped in 693a78d + 7ec0b35 with unit
coverage over a fabricated principal. What a fabricated principal cannot show is
that the group names the shared directory actually returns -- short RDN values,
not DNs -- are the ones Dashboard:GroupToTag keys match. Two [LiveLdapFact]s
close that: gw-viewer binds for real, its GwReader membership grants team-a, and
IDashboardSessionAcl then admits a team-a-tagged session and refuses a
team-b-tagged one; multi-role takes the Administrator bypass. The mapping is
config-side only -- no GLAuth entry, group, or membership was added, and
glauth.md records that explicitly so a future reader does not go looking for a
directory change that never happened.
multi-role is a member of GwReader as well as GwAdmin, so it holds team-a too.
Its bypass is therefore asserted on team-b and on the untagged session -- the two
it would lose if the Administrator branch were ever dropped -- rather than on
team-a, which would pass either way.
One cheap hardening from a prior review: a GatewayOptionsTests case binds
Dashboard:GroupToTag through a real ConfigurationBuilder and looks the group up
mis-cased. The property initializer seeds an OrdinalIgnoreCase dictionary, but
only the binder decides whether that instance survives; if it did not, a
mis-cased group name from the directory would grant no tags and the ACL would
deny with no diagnostic.
Docs follow the shipped shape: docs/Sessions.md gains the session-tag model
(owner-key sourced, immutable, visibility-not-access), gateway.md and CLAUDE.md
gain the ACL in their dashboard-auth paragraphs, and three
GatewayDashboardDesign.md passages that still described the ACL as outstanding
now describe both gated seams and the decision order. GatewayConfiguration.md's
ShowTagValues row no longer claims the redaction is the only thing between a
Viewer and another session's values -- it is now the second of two independent
layers. gateway.md's hub-token lifetime corrected 30 minutes -> 5, matching
HubTokenService. Authentication.md disambiguates --dashboard-tags as the only
constraint flag that splits on commas. The plan doc header is Implemented; its
as-built section 12 already existed and is not duplicated.
Verified: NonWindows.slnx builds clean; GatewayOptions/DashboardSessionAcl/
EventsHub filters 37/37; the live-LDAP suite skips cleanly without the env var
and runs 7/7 green against the shared GLAuth with it.
Groundwork for the per-session dashboard event ACL (docs/plans/2026-07-10-dashboard-session-acl-tst15.md 3.2): a dashboard group can now grant visibility tags, and untagged sessions default to AdminOnly. Enforcement lands with the EventsHub ACL; nothing consumes the grant yet.
GroupToTag is deliberately uncoupled from GroupToRole - a group may appear in either map, both, or neither - and is validated for shape only. Tags gate dashboard event visibility, never data access.
Auth 0.1.5 -> 0.2.0, Health 0.2.0 -> 0.3.0, Secrets/.Abstractions/.Ui
0.6.1 -> 0.6.2. Theme, GalaxyRepository, Audit, Configuration, Telemetry
and Telemetry.Serilog were already at the newest version on the feed.
Checked against the shared-lib source rather than the version numbers,
because these packages are versioned as a family and a bump is not by
itself evidence that the package changed:
- Auth 0.2.0 is the only one carrying content for us: LDAP backup-DC
failover (FallbackServers, endpoint walk with sticky preference,
boot-time entry validation). Purely additive; the default is empty,
which leaves single-endpoint behaviour unchanged.
- Health 0.3.0 carries a breaking change, but every line of it is in
ZB.MOM.WW.Health.Akka, which we do not reference. No commit touched
the core ZB.MOM.WW.Health package between 0.2.0 and 0.3.0.
- Secrets 0.6.2 is a message-only change: one validator string literal
gains mounted-volume guidance. SecretsStorePathRules is untouched.
The four non-csproj files are not a separate feature. Configuration/
LdapOptions is a deliberate shadow of the shared type and carries an
explicit warning to mirror any new upstream field, because AddZbLdapAuth
binds the whole MxGateway:Ldap section onto the shared options. So
FallbackServers is live on our config surface the moment the package
lands, and without the mirror an operator could configure a backup DC
that works but is invisible on the dashboard's Settings page. The
Settings row renders "none" when empty, since that is the answer someone
who believes a backup DC is configured actually needs.
Entry syntax is deliberately NOT re-validated here: the shared validator
already fails the boot on a malformed entry and owns the (internal)
parser, so a second copy would drift. Note both validators skip entirely
when Ldap:Enabled is false.
Verified the binder is non-strict (ErrorOnUnknownConfiguration is unused
anywhere in the tree), so the upgrade could not break startup on a
newly-recognised key either way.
Build 0 warnings / 0 errors; gateway suite 892/892, unchanged. The live
LDAP tests are opt-in and were not run, so the failover path itself is
covered only by the shared library's own tests.
Rooted is not the same as safe, and the gap between the two cost a production
host every one of its API keys on 2026-08-09.
MxGateway:Authentication:SqlitePath was set to an absolute path inside the
directory the upgrade procedure renames to Server.bak.*. That passes the
existing rooted check cleanly. The deploy renamed the directory away, the store
went with it, and the gateway created a fresh empty one at the same path — no
error, no log line. No gRPC consumer could authenticate for two days. The deploy
itself was correct: the binaries were the point of the rename and the store was
collateral.
GatewayConfigPathRules gains AddIfUnderContentRoot, applied to the auth store
and the Galaxy snapshot. Both are written by the running process and both are
lost the same way. The rule compares resolved full paths and requires a
directory-separator boundary, so a sibling directory whose name merely starts
with the content root's ("/srv/app-data" against "/srv/app") is not treated as
inside it — on a fail-closed startup rule, that false positive would be a
gateway that refuses to boot on a legitimate path. Case sensitivity follows the
running OS rather than assuming case-insensitivity everywhere, which would
reject /srv/App as under /srv/app on Linux where they are different directories.
The rule is not exempted in Development. An environment-conditional guard is
never exercised where the mistake is made, and what failed in production was a
config that looked fine.
Secrets:SqlitePath is the same defect one layer down: it shipped as a bare
relative "mxgateway-secrets.db", which is how a stray database landed in
src/…Server/ and tripped the repository's tree-hygiene test. It is bound by the
shared ZB.MOM.WW.Secrets package, so appsettings.json now ships no value and the
default is computed from CommonApplicationData in code — the same mechanism
SEC-33 already used for the Galaxy snapshot, ten lines away, for the same reason.
Setting a default for an unset key is deliberately not the same act as
relocating a value someone configured, which these rules still refuse to do.
Note the migration edge this creates: a host relying on the old repo default now
looks somewhere new, finds nothing, and creates an empty store — this bug
re-introduced by its own fix. Deployed hosts are safe because they set the path
explicitly, in appsettings copied forward or in the service environment. The
latter is the more robust of the two, since it cannot be lost by a missed
preserve step.
The pipe name mxaccess-gateway-{pid}-session-{32hex} plus .NET's
CoreFxPipe_ prefix overflowed the 104-byte Unix-domain-socket path limit
under the default per-user macOS TMPDIR (~49 chars), so every test that
opened a real pipe threw ArgumentOutOfRangeException at pipe creation
unless TMPDIR=/tmp was exported. Rename to mxgw-{pid}-{sessionUid} (the
session guid hex without the session- prefix; worst-case 43 chars) and
shorten the three test-fixture names the same way. Uniqueness is
unchanged: gateway pid + full session guid. The worker receives the pipe
name via its launch command line, so mixed Server/Worker deploy SHAs are
unaffected. Docs updated in the same change (gateway.md,
GatewayProcessDesign, GatewayConfiguration, Sessions, CLAUDE.md); new
regression test pins the format and the length budget.
Verified: SessionManagerTests 39/39; SessionWorkerClientFactory,
GatewayEndToEndFakeWorkerSmoke, WorkerClient, and ReconnectReplay suites
33/33 under the default macOS TMPDIR — this also retires the
previously-misdiagnosed 'macOS pipe-timeout test failures': they were
this path-length throw, not a timeout-message defect.
SEC-36's primary check -- dashboard /login through the real DashboardAuthenticator
search bind -- was deferred because windev's gateway was crash-looping on the stale
deployment filed as NEXT-07. That host was redeployed 2026-08-07, so the check ran:
login as multi-role returns 302 with the dashboard cookie and the authenticated page
renders the admin nav, while an anonymous control still redirects to /login. The
rotated service-account credential is now proven end-to-end on the deployed host, not
only by the equivalent ldapsearch primitive, and the runbook's Correction 3 is past
tense throughout rather than describing a fault that no longer exists.
Also record why windev runs the Staging environment name. The redeploy tripped SEC-06's
Production hard-stop on Ldap:Transport=None, and windev cannot satisfy it: it binds the
shared GLAuth, which offers no TLS, and runs Dashboard:DisableLogin=true. The Production
label contradicted its own configuration, so the host was relabelled rather than the
guard weakened -- exactly the permissive-staging-rig case the SEC-35 section already
carves out.
Repo-side half of SEC-36. The appsettings.json plaintext was already discharged
before this branch (HEAD ships the fail-closed ${secret:ldap/mxgateway/bind}
store reference), so the residual leak was the literal value in glauth.md,
docs/GatewayTesting.md, and the historical archreview SEC-06 evidence -- all
scrubbed to <service-account-password> placeholders pointing at the source of
truth scadaproj/infra/glauth/.
- csproj: add <UserSecretsId>mxaccessgw-server</UserSecretsId> (dev channel)
- GatewayOptionsValidator: blank-password message now names both channels
(dev user-secrets, deployed MxGateway__Ldap__ServiceAccountPassword)
- test: assert the message names both channels
- docs: GatewayConfiguration.md (three channels + rotation note), glauth.md
(placeholders + rotation-required + runbook pointer), GatewayTesting.md
- new operator runbook docs/runbooks/SEC-36-ldap-credential-rotation.md
(live rotation + NSSM staging remain operator-pending)
- tracking: SEC-36 -> Done (repo-side) in both registers + change-log
Deviation: kept the ${secret:} reference in appsettings.json rather than
deleting it (spec step 2 assumed the stale plaintext baseline); deleting it
would regress the shipped/documented/tested secret-store channel.
git grep -i for the old value is empty across all tracked files.
TST-27: docs/GatewayConfiguration.md's ShowTagValues row no longer says
"Reserved" — it now states what false (default) does (DashboardEventBroadcaster
blanks tag values from a deep-cloned MxEvent before the SignalR events-hub
mirror), the security relevance (no per-session hub ACL yet, so this
redaction is the only thing between a low-trust Viewer and other sessions'
tag values), and the honest scope limit (does not cover /browse).
WRK-26 (discharges IPC-29): docs/MxAccessWorkerInstanceDesign.md's "Outbound
Queues" section rewritten from the stale five-level priority list to the
two-class Control/Event scheduler actually shipped, with the collapsed-
decision rationale, and the overflow paragraph rewritten to the implemented
fail-fast. docs/WorkerFrameProtocol.md gained a "Write Scheduling And
Sequencing" section describing HEAD truthfully: WRK-23's peek-stamp-commit
sequencing is live, WRK-25's event-batch flush coalescing is not (the drain
loop still awaits each event write individually), and WRK-22's cancellation
tombstone is not yet defined (noted as pending, not documented as shipped).
CLI-42: clients/rust/README.md and docs/ClientPackaging.md document the
vendored Rust proto layout matching build.rs — repo-path-first resolution
falling back to clients/rust/protos/, the check-codegen.ps1 Check 3 refresh
rule, and why cargo package/publish run without --no-verify.
CLI-43: docs/style-guides/JavaStyleGuide.md now says Java 17 (Ignition 8.3
baseline), mirroring CLI-12's wording, matching the shipped build.gradle.
IPC-28: docs/Grpc.md's exception-mapping prose gained CommandTooLarge ->
ResourceExhausted, and the Invoke section gained the oversized-payload
sentence, cross-referencing GatewayConfiguration.md's headroom rule.
Tracking: TST-27, WRK-26, CLI-42, CLI-43, IPC-28 flipped to Done and IPC-29
marked discharged-by-WRK-26 in 00-tracking.md and the 20/30/50/60 domain
registers, with a 2026-08-07 change-log entry.
Doc-only change; no source, proto, or test edits.
SEC-33: make rooting host-meaningful and stop shipping foreign-platform literals.
- Delete IsRootedForAnyPlatform; AddIfNotRooted now uses Path.IsPathRooted (current OS).
- Promote AddIfNotRooted/AddIfInvalidPath to shared GatewayConfigPathRules so the new
Galaxy validator reuses them and the two validators cannot drift.
- Remove Authentication:SqlitePath and Galaxy:SnapshotCachePath Windows literals from
appsettings.json; the CommonApplicationData-derived code defaults take over. The
Galaxy default is seeded as a configuration value before AddZbGalaxyRepository
(SnapshotCachePath is init-only, so a PostConfigure mutation cannot compile).
- New GalaxyRepositoryOptionsValidator (ValidateOnStart) enforces a valid, host-rooted
SnapshotCachePath when PersistSnapshot is true.
- Root-cause the stray junk-named auth DB: host start eagerly builds
AuthSqliteConnectionFactory; under the non-rooted Windows literal on macOS SQLite
wrote it relative to the test bin CWD. The three real-host-start tests now pin
SqlitePath to a temp path.
SEC-34: verification cache Invalidate-vs-in-flight-repopulation race closed with a
per-key generation counter (bump-before-evict, snapshot-then-recheck). The expiry
cap (window 2) takes the documented fallback: the library verification identity
carries no ExpiresUtc, so the cache cannot cap at the key's expiry (donor-library ask).
GWC-24 rider: cap MxGateway:Events:QueueCapacity at int.MaxValue/2 so the derived
checked(2 * EventChannelCapacity) in WorkerClient cannot overflow at session creation.
SEC-35 (doc-only): note IsProduction() env-name semantics in GatewayConfiguration.md.
Docs updated same commit (GatewayConfiguration.md, Authentication.md) and tracking
registers/change-log flipped (00-tracking.md, 40-security-dashboard.md).