11 Commits

Author SHA1 Message Date
Joseph Doherty a87c05754a docs(deploy): PROD CVD reactor content 2026-11-22 (9 instances on site-par, 7 API methods, BTDB, keys)
Claude-Session: https://claude.ai/code/session_01GT67AD5mc2BnaaAfY3hqqQ
2026-11-22 12:06:09 -05:00
Joseph Doherty dc87289faf docs(deploy): PROD site-app (id 2) co-located on app-vp03/vp04, MxGateway connection id 2 Connected
Claude-Session: https://claude.ai/code/session_01GT67AD5mc2BnaaAfY3hqqQ
2026-11-20 17:43:47 -05:00
Joseph Doherty 3ca968e26b docs(deploy): PROD site-par MxGateway connection re-pointed to the PAR gateways (not installed yet)
Claude-Session: https://claude.ai/code/session_01GT67AD5mc2BnaaAfY3hqqQ
2026-11-20 17:23:23 -05:00
Joseph Doherty 530eaf32c1 docs(deploy): PROD cluster config - Site site-par (id 1), mxgw key scadabridge-prod shared vp03/vp04, MxGateway connection id 1 Connected
Claude-Session: https://claude.ai/code/session_01GT67AD5mc2BnaaAfY3hqqQ
2026-11-20 16:37:41 -05:00
Joseph Doherty facf24b295 docs(deploy): PROD 2e0ce81e installed on vp03/vp04 (stopped, gated on DB indexes) + site-par pair running on par-vp03/04
Claude-Session: https://claude.ai/code/session_01GT67AD5mc2BnaaAfY3hqqQ
2026-11-20 15:39:40 -05:00
Joseph Doherty 9109e4e905 docs(deploy): QA vq03/vq04 on 2e0ce81e (#144/#145) + NAM AD login, one window
Claude-Session: https://claude.ai/code/session_01GT67AD5mc2BnaaAfY3hqqQ
2026-11-19 16:13:55 -05:00
Joseph Doherty 98a8df1f9d deploy(vd03): 2e0ce81e (#144/#145) + NAM AD login on DEV 2026-11-19 15:17:16 -05:00
Joseph Doherty 2e0ce81ea0 Merge branch 'fix/145-management-zero-role': refuse zero-role principals on management/CLI surfaces (#145)
Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
2026-11-17 11:48:03 -05:00
Joseph Doherty a5243b1e87 docs(security): no role, no login now covers the management/CLI surfaces (#145)
Component-Security "No role, no login": replace the "HTTP Basic is not covered
yet" note with the Basic and Bearer behaviour, the throttle and log details;
bearer section notes a role-less token is refused. Component-CLI: a zero-role
user sees the bad-password 401. Component-ManagementService: 401 covers it.
CLAUDE.md Security & Auth bullet updated.

Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
2026-11-17 11:45:51 -05:00
Joseph Doherty 1d2170a152 fix(management): refuse zero-role principals on the management/CLI surfaces (#145)
ManagementAuthenticator — the only identity source for POST /management, the
audit REST endpoints and the debug-stream hub — now enforces "no role, no
login" the way #144 does for /auth/login and /auth/token:

- HTTP Basic: a bind whose groups map to zero roles, or whose role mapping
  throws, answers the bad-password 401 AUTH_FAILED body and counts as a
  LoginThrottle failure. RecordSuccess moved after the mapping, so a refusal
  no longer clears earlier failures. Request cancellation still propagates.
- Bearer: a token with no role claim (minted before #144 shipped) gets the
  invalid-token 401; no throttle, as for every bearer failure.
- Each refusal is logged at Warning (category ManagementAuthenticator) with
  the request path, the username and the reason.

Before this, a zero-role Basic caller reached every command that
ManagementActor.GetRequiredRoles opens to any authenticated user.
DisableLogin is unchanged.

Tests: ManagementZeroRoleRefusalTests — zero roles / mapper throws / one role
over Basic and Bearer for /management (ListSites), /api/audit/query and the
debug-stream hub; every null-role command is unreachable by a zero-role
principal; throttle accounting. 47 of the 58 fail against the old
authenticator (the 11 that pass are the one-role/bypass regression guards).

Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
2026-11-17 11:45:51 -05:00
Joseph Doherty d48e1667f1 Merge branch 'fix/144-refuse-zero-role-login': refuse zero-role sign-in and sessions (#144)
Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
2026-11-17 11:29:48 -05:00
15 changed files with 1442 additions and 11 deletions
+1 -1
View File
@@ -195,7 +195,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
- Authentication: direct LDAP bind (username/password), no Kerberos/NTLM. LDAPS/StartTLS required.
- Two session models: the **UI** uses an HttpOnly/Secure cookie carrying a data-protected claims ticket (no embedded JWT; shared Data Protection keys; 30-minute idle timeout, DB-only role re-map every 15 min); the **CLI/API** exchanges credentials once per invocation at `POST /auth/token` for a 15-minute HS256 **bearer** JWT (`JwtSigningKey`, >= 32 bytes, shared by both centrals; `management_bearer: true` capability flag; Basic stays as fallback). Bearer roles are authoritative until expiry (revocation lag <= `JwtExpiryMinutes`; rotating the key kills every token). Cookies are the correct transport for Blazor Server (SignalR circuits). Spec: `Component-Security.md`.
- LDAP failure: new logins fail; active sessions continue with current roles.
- **No role, no login (ScadaBridge#144, 2026-11-17).** `/auth/login` and `/auth/token` refuse a bind whose groups map to zero roles (or whose role mapping throws) with the exact bad-password response + a throttle failure; the reason is logged server-side only. A cookie session whose refresh maps to zero roles is ended (`CookieSessionValidator` → Reject). HTTP Basic on the management surfaces is not covered yet (follow-up). Spec: `Component-Security.md` *No role, no login*.
- **No role, no login (ScadaBridge#144/#145, 2026-11-17).** `/auth/login` and `/auth/token` refuse a bind whose groups map to zero roles (or whose role mapping throws) with the exact bad-password response + a throttle failure; the reason is logged server-side only. A cookie session whose refresh maps to zero roles is ended (`CookieSessionValidator` → Reject). The management/CLI surfaces (`/management`, `/api/audit/*`, `/hubs/debug-stream`) refuse the same way in `ManagementAuthenticator` (ScadaBridge#145): a zero-role or mapper-fault Basic bind gets the bad-password 401 + a throttle failure, and a bearer token with no role claim gets the invalid-token 401. Spec: `Component-Security.md` *No role, no login*.
- Load balancer in front of central UI; cookie-embedded JWT + shared Data Protection keys for failover transparency.
- Two-person MxGateway secured writes (M7): two new global roles — `Operator` (initiates) + `Verifier` (approves) — added alongside the canonical `Administrator`/`Designer`/`Deployer`/`Viewer`, with `RequireOperator`/`RequireVerifier` policies. An Operator submits a secured write from the Central UI Secured Writes page (`/operations/secured-writes`); it stays a `Pending` `PendingSecuredWrite` row until a *distinct* Verifier approves it (no-self-approval enforced server-side in the ManagementActor, plus a compare-and-swap race guard). Approval relays a `WriteTagRequest` to the site MxGateway; MxGateway-protocol connections only; each lifecycle event (submit/approve/reject/execute) emits a best-effort `AuditChannel.SecuredWrite` / `AuditKind.SecuredWrite*` central-direct-write row sharing the row id as `CorrelationId`. (SecuredWrite audit rows stamp `SourceNode` via `ICentralAuditWriter`/`INodeIdentityProvider`.) Pending secured writes expire server-side after a configurable TTL (`ManagementServiceOptions.SecuredWritePendingTtl`, default 24 h): an overdue `Pending` row is CAS'd to `Expired` (never relayed) — enforced at approve/reject and swept opportunistically on list (arch-review S2, `AuditKind.SecuredWriteExpire`).
- **Inbound API key store — per node by default, shared on central pairs (2026-09-02, register row 108).** The shared `ZB.MOM.WW.Auth.ApiKeys` library stores inbound keys in **SQLite per node** unless `ScadaBridge:InboundApi:ApiKeyStore:Provider=SqlServer`, which points both central nodes at the same `zbauth` schema in the configuration database (library `ZB.MOM.WW.Auth.ApiKeys.SqlServer` 0.3.0) — under `SqlServer`, create/revoke/rotate/scope changes are cluster-wide immediately; a node's existing SQLite file is imported once at boot (insert-where-absent, safe on both nodes of a pair) and renamed `*.imported`. Central pairs **MUST** run `Provider=SqlServer` — a clustered Central left on `Sqlite` boots fine but `StartupValidator` logs a Warning, because a failover otherwise 401s every inbound client until keys are re-issued on the survivor. Both docker rigs (`docker/`, `docker-env2/`) run `SqlServer`; QA (`vq03`/`vq04`) runs `SqlServer` for both inbound keys and secrets since 2026-10-22 (deployment record `2026-10-22-wonder-app-vq03-vq04-3808b49f.md`); DEV (`vd03`, single node — not production) is not switched. The boot import's `*.imported` rename needs the importer's pooled SQLite handle released first — Windows refuses to rename an open file (fixed `6ed6800b`; pre-fix Windows builds need the manual rename in the runbook). Design: `docs/plans/2026-09-02-inbound-api-key-shared-store-design.md`; runbook: `docs/operations/inbound-api-key-reissue.md` → *Central pairs*.
@@ -0,0 +1,159 @@
# ScadaBridge DEV (wonder-app-vd03) 061432c8 -> 2e0ce81e : ScadaBridge#144 + #145 (no role, no login). Binary-only: NO migration, NO env change.
# winrm --timeout 1200 ps vd03 -f dev-deploy.ps1
# Single-node box: ONE central + ONE site; DB on this host.
# Order: preflight gates -> reg export -> DB backup (safety, COPY_ONLY) -> keep appsettings
# -> stop Site, Central -> failureflag -> swap app/cli -> restore appsettings -> start Central, Site -> post checks.
# On failure after the stop: old binaries back, restart. Values are never printed.
$ErrorActionPreference = 'Stop'
$ProgressPreference = 'SilentlyContinue'
$sha = '2e0ce81e'
$fromSha = '061432c8'
$zipSha256 = '6C706B0A76B7E9EC7CA00E12CEA19C5D86814BEAF66B541F527CBFC3D6572E23'
$stageDir = "E:\ApiInstall\_stage-sb-$sha"
$stage = "$stageDir\unpacked"
$root = 'E:\ApiInstall\ScadaBridge'
$bakDir = 'E:\ApiInstall\ScadaBridge-backups'
$ts = Get-Date -Format 'yyyyMMddTHHmmss'
$log = "$root\_DEPLOY-NOTES-$sha-$ts.txt"
$settings = 'appsettings.json','appsettings.Central.json','appsettings.Site.json'
$centralKey = 'HKLM:\SYSTEM\CurrentControlSet\Services\ScadaBridge-Central'
function Say($m) { $line = "[{0:HH:mm:ss.fff}] {1}" -f (Get-Date), $m; $line; Add-Content -Path $log -Value $line }
function Is-Retired($e) { $e -like 'AuditLog__HashChain__*' -or $e -like 'AuditLog__MaintenanceConnectionString=*' }
function Digest($entries) {
$h = [System.Security.Cryptography.SHA256]::Create(); $o = @()
foreach ($e in $entries) { $i = $e.IndexOf('='); if ($i -lt 1) { $o += '<unparsable>'; continue }
$v = $e.Substring($i+1); $x = -join ($h.ComputeHash([Text.Encoding]::UTF8.GetBytes($v)) | % { $_.ToString('x2') })
$o += ("{0} sha256:{1} len={2}" -f $e.Substring(0,$i), $x.Substring(0,12), $v.Length) }
$h.Dispose(); $o | Sort-Object
}
function Wait-Ready($url, $seconds) {
$deadline = (Get-Date).AddSeconds($seconds)
while ((Get-Date) -lt $deadline) { try { if ((Invoke-WebRequest -UseBasicParsing $url -TimeoutSec 5).StatusCode -eq 200) { return $true } } catch {}; Start-Sleep 2 }
return $false
}
function Stop-Both {
foreach ($s in 'ScadaBridge-Site','ScadaBridge-Central') {
if ((Get-Service $s).Status -ne 'Stopped') { Say "stopping $s"; Stop-Service $s -Force; (Get-Service $s).WaitForStatus('Stopped','00:02:00') }
Say "$s stopped"
}
$deadline = (Get-Date).AddSeconds(30)
while ((Get-Process -Name 'ZB.MOM.WW.ScadaBridge.Host' -ErrorAction SilentlyContinue) -and (Get-Date) -lt $deadline) { Start-Sleep 1 }
if (Get-Process -Name 'ZB.MOM.WW.ScadaBridge.Host' -ErrorAction SilentlyContinue) { throw "Host.exe still running after stop" }
Say "no Host.exe process remains"
}
function Start-Both {
Say "starting ScadaBridge-Central"; Start-Service 'ScadaBridge-Central'
$ok = Wait-Ready 'http://localhost:8085/health/ready' 180; Say "central ready=$ok"
if (-not $ok) { throw "central did not become ready" }
Say "starting ScadaBridge-Site"; Start-Service 'ScadaBridge-Site'
$ok = Wait-Ready 'http://localhost:8084/health/ready' 180; Say "site ready=$ok"
if (-not $ok) { throw "site did not become ready" }
}
Say "ScadaBridge#144/#145 deploy $fromSha -> $sha on $env:COMPUTERNAME"
# ---- preflight gates (nothing changed yet) ----------------------------------------------------
if ((Get-FileHash "$stageDir\scadabridge-$sha.zip" -Algorithm SHA256).Hash -ne $zipSha256) { throw "zip hash mismatch" }
if (Test-Path $stage) { Remove-Item $stage -Recurse -Force }
Expand-Archive "$stageDir\scadabridge-$sha.zip" $stage
$stagedVer = (Get-Item "$stage\app\ZB.MOM.WW.ScadaBridge.Host.exe").VersionInfo.ProductVersion
$stagedCli = (Get-Item "$stage\cli\scadabridge.exe").VersionInfo.ProductVersion
if ($stagedVer -notlike "*+$sha*" -or $stagedCli -notlike "*+$sha*") { throw "staged exe is $stagedVer / $stagedCli" }
$liveVer = (Get-Item "$root\app\ZB.MOM.WW.ScadaBridge.Host.exe").VersionInfo.ProductVersion
if ($liveVer -notlike "*+$fromSha*") { throw "live exe is $liveVer, not $fromSha" }
Say "staged $stagedVer; live $liveVer"
Say ("staged app files {0}, cli files {1}" -f (Get-ChildItem "$stage\app" -Recurse -File).Count, (Get-ChildItem "$stage\cli" -Recurse -File).Count)
$envBefore = @((Get-ItemProperty $centralKey).Environment)
$retired = @($envBefore | Where-Object { Is-Retired $_ })
Say ("central env entries {0}; retired keys present: {1}" -f $envBefore.Count, $retired.Count)
$cs = ($envBefore | Where-Object { $_ -like 'ScadaBridge__Database__ConfigurationDb=*' }) -replace '^[^=]*=',''
$sb = New-Object System.Data.SqlClient.SqlConnectionStringBuilder $cs
$db = $sb['Database']; if (-not $db) { $db = $sb.InitialCatalog }
if ($db -ne 'ScadaBridge') { throw "unexpected catalog" }
$env:SQLCMDPASSWORD = $sb.Password
function Sql-Scalar($q) { $r = & sqlcmd -S $sb.DataSource -d $db -U $sb.UserID -C -h -1 -W -b -Q "SET NOCOUNT ON; $q"; if ($LASTEXITCODE -ne 0) { throw "sql failed: $q" }; ($r | Where-Object { $_ -and $_.Trim() } | Select-Object -First 1).Trim() }
$chainQ = "SELECT CONCAT((SELECT COUNT(*) FROM __EFMigrationsHistory),'|',COL_LENGTH('dbo.AuditLog','ChainSeq'),'|',COL_LENGTH('dbo.AuditLog','RowHash'),'|',CASE WHEN OBJECT_ID('dbo.AuditLogChainHead','U') IS NULL THEN 0 ELSE 1 END,'|',CASE WHEN OBJECT_ID('dbo.AuditLogChainTombstone','U') IS NULL THEN 0 ELSE 1 END,'|',(SELECT COUNT(*) FROM sys.indexes WHERE object_id=OBJECT_ID('dbo.AuditLog') AND name IN ('IX_AuditLog_ChainSeq','IX_AuditLog_Unchained')),'|',CASE WHEN DATABASE_PRINCIPAL_ID('scadabridge_audit_chainer') IS NULL THEN 0 ELSE 1 END,'|',CASE WHEN DATABASE_PRINCIPAL_ID('scadabridge_audit_purger') IS NULL THEN 0 ELSE 1 END,'|',CASE WHEN DATABASE_PRINCIPAL_ID('scadabridge_audit_writer') IS NULL THEN 0 ELSE 1 END,'|',(SELECT COUNT_BIG(*) FROM dbo.AuditLog))"
$pre = Sql-Scalar $chainQ
Say "schema before (migrations|ChainSeq|RowHash|Head|Tomb|chainIdx|chainer|purger|writer|rows): $pre"
if (-not $pre.StartsWith('65|')) { throw "expected 65 migrations: $pre" }
# ---- backups (services still running) ---------------------------------------------------------
foreach ($s in 'ScadaBridge-Central','ScadaBridge-Site') {
& reg.exe export "HKLM\SYSTEM\CurrentControlSet\Services\$s" "$stageDir\_envbak-$s-$ts.reg" /y | Out-Null
if ($LASTEXITCODE -ne 0) { throw "reg export $s failed" }
Say "reg export $s -> $stageDir\_envbak-$s-$ts.reg"
}
New-Item -ItemType Directory -Force -Path $bakDir | Out-Null
$bak = "$bakDir\ScadaBridge_pre-$($sha)_$ts.bak"
& sqlcmd -S $sb.DataSource -d master -U $sb.UserID -C -b -Q "BACKUP DATABASE [$db] TO DISK = N'$bak' WITH COPY_ONLY, CHECKSUM, COMPRESSION, INIT" | Out-Null
if ($LASTEXITCODE -ne 0) { throw "backup failed" }
& sqlcmd -S $sb.DataSource -d master -U $sb.UserID -C -b -Q "RESTORE VERIFYONLY FROM DISK = N'$bak' WITH CHECKSUM" | Out-Null
if ($LASTEXITCODE -ne 0) { throw "verifyonly failed" }
Say "db backup $bak ($((Get-Item $bak).Length) bytes, VERIFYONLY ok)"
$keep = New-Item -ItemType Directory -Force -Path "$root\_appsettings-keep-$ts"
foreach ($f in $settings) { Copy-Item "$root\app\$f" "$keep\$f"; Say ("keep {0} sha256 {1}" -f $f, (Get-FileHash "$keep\$f" -Algorithm SHA256).Hash) }
# ---- outage window -------------------------------------------------------------------------
$swapped = $false; $migrated = $false; $envChanged = $false
try {
Stop-Both
foreach ($s in 'ScadaBridge-Central','ScadaBridge-Site') {
& sc.exe failureflag $s 1 | Out-Null; if ($LASTEXITCODE -ne 0) { throw "sc failureflag $s failed" }
}
Rename-Item "$root\app" "app.bak.$ts"; Rename-Item "$root\cli" "cli.bak.$ts"; $swapped = $true
Say "renamed app/cli -> *.bak.$ts"
Copy-Item "$stage\app" "$root\app" -Recurse
Copy-Item "$stage\cli" "$root\cli" -Recurse
foreach ($f in $settings) {
Copy-Item "$keep\$f" "$root\app\$f" -Force
if ((Get-FileHash "$keep\$f" -Algorithm SHA256).Hash -ne (Get-FileHash "$root\app\$f" -Algorithm SHA256).Hash) { throw "appsettings restore mismatch: $f" }
Say "restored $f (sha256 match)"
}
Say ("live Host " + (Get-Item "$root\app\ZB.MOM.WW.ScadaBridge.Host.exe").VersionInfo.ProductVersion + " / CLI " + (Get-Item "$root\cli\scadabridge.exe").VersionInfo.ProductVersion)
Start-Both
}
catch {
Say "ERROR: $($_.Exception.Message)"
Say "AUTO-ROLLBACK (migrated=$migrated swapped=$swapped envChanged=$envChanged)"
try {
Stop-Both
if ($migrated) {
& sqlcmd -S $sb.DataSource -d master -U $sb.UserID -C -b -Q "ALTER DATABASE [$db] SET SINGLE_USER WITH ROLLBACK IMMEDIATE; RESTORE DATABASE [$db] FROM DISK = N'$bak' WITH REPLACE, CHECKSUM; ALTER DATABASE [$db] SET MULTI_USER;" | ForEach-Object { Say " restore: $_" }
if ($LASTEXITCODE -ne 0) { throw "DB restore failed" }
Say "DB restored from $bak"
}
if ($swapped) {
if (Test-Path "$root\app") { Rename-Item "$root\app" "app.failed.$ts" }
if (Test-Path "$root\cli") { Rename-Item "$root\cli" "cli.failed.$ts" }
Rename-Item "$root\app.bak.$ts" 'app'; Rename-Item "$root\cli.bak.$ts" 'cli'
Say "restored app/cli from *.bak.$ts"
}
if ($envChanged) { Set-ItemProperty -Path $centralKey -Name Environment -Value ([string[]]$envBefore) -Type MultiString; Say "central env restored" }
Start-Both
Say ("rolled back; live Host " + (Get-Item "$root\app\ZB.MOM.WW.ScadaBridge.Host.exe").VersionInfo.ProductVersion)
} catch { Say "AUTO-ROLLBACK FAILED: $($_.Exception.Message) - manual action needed" }
Remove-Item Env:SQLCMDPASSWORD -ErrorAction SilentlyContinue
exit 1
}
# ---- post checks (verify of this deploy) ------------------------------------------------------
Start-Sleep 20
$q2 = Sql-Scalar "SELECT CONCAT(COUNT_BIG(*),'|',CONVERT(varchar(33),MAX(OccurredAtUtc),126)) FROM dbo.AuditLog"
Say "auditlog rows|newest: $q2"
$logDir = "$root\central\logs"
if (Test-Path $logDir) {
$f = Get-ChildItem $logDir -File | Sort-Object LastWriteTime -Descending | Select-Object -First 1
$hits = @(Select-String -Path $f.FullName -Pattern 'no longer supported' -SimpleMatch)
Say ("central log {0}: retired-key warnings since start = {1}" -f $f.Name, $hits.Count)
}
foreach ($s in 'ScadaBridge-Central','ScadaBridge-Site') { Say ("$s " + (Get-Service $s).Status + " / " + ((& sc.exe qfailureflag $s | Out-String) -replace '\s+',' ').Trim()) }
Remove-Item Env:SQLCMDPASSWORD -ErrorAction SilentlyContinue
Say "DEPLOY DONE ts=$ts backup=$bak"
exit 0
@@ -0,0 +1,131 @@
# ScadaBridge QA (vq03/vq04) 061432c8 -> 2e0ce81e (#144/#145, no migration) + NAM AD LDAP, one window. Rendered per call: ACTION / TS.
# prep | backup (vq03) | stop-site | stop-central | swap | start-central | start-site | verify | rollback-swap
# Derived from ScadaBridge deploy-prep/2026-11-15-143/qa.tmpl.ps1 (migrate dropped; the swap also installs the per-node NAM
# appsettings.Central.json and removes the GLAuth password env entry from the Central service).
$ErrorActionPreference = 'Stop'
$ProgressPreference = 'SilentlyContinue'
$Action = '__ACTION__'
$ts = '__TS__'
$sha = '2e0ce81e'
$fromSha = '061432c8'
$zipSha256 = '6C706B0A76B7E9EC7CA00E12CEA19C5D86814BEAF66B541F527CBFC3D6572E23'
$stageDir = "E:\psrem-stage\sb-$sha"
$stage = "$stageDir\unpacked"
$root = 'E:\ApiInstall\ScadaBridge'
$log = "$root\_DEPLOY-NOTES-$sha-$ts.txt"
$settings = 'appsettings.json','appsettings.Central.json','appsettings.Site.json'
$centralKey = 'HKLM:\SYSTEM\CurrentControlSet\Services\ScadaBridge-Central'
$namCentral = "$stageDir\appsettings.Central.json.nam-" + $env:COMPUTERNAME.Substring($env:COMPUTERNAME.Length-4).ToLower()
$bak = "F:\MSSQL\BACKUP\ScadaBridge_pre-$($sha)_$ts.bak"
function Say($m) { $line = "[{0:HH:mm:ss.fff}] {1} {2}" -f (Get-Date), $env:COMPUTERNAME, $m; $line; Add-Content -Path $log -Value $line }
function Is-Dropped($e) { $e -like 'ScadaBridge__Security__Ldap__ServiceAccountPassword=*' }
function Digest($entries) {
$h = [System.Security.Cryptography.SHA256]::Create(); $o = @()
foreach ($e in $entries) { $i = $e.IndexOf('='); if ($i -lt 1) { $o += '<unparsable>'; continue }
$v = $e.Substring($i+1); $x = -join ($h.ComputeHash([Text.Encoding]::UTF8.GetBytes($v)) | % { $_.ToString('x2') })
$o += ("{0} sha256:{1} len={2}" -f $e.Substring(0,$i), $x.Substring(0,12), $v.Length) }
$h.Dispose(); $o | Sort-Object
}
function Wait-Ready($url, $seconds) {
$deadline = (Get-Date).AddSeconds($seconds)
while ((Get-Date) -lt $deadline) { try { if ((Invoke-WebRequest -UseBasicParsing $url -TimeoutSec 5).StatusCode -eq 200) { return $true } } catch {}; Start-Sleep 2 }
return $false
}
function Stop-Svc($s) {
if ((Get-Service $s).Status -ne 'Stopped') { Say "stopping $s"; Stop-Service $s -Force; (Get-Service $s).WaitForStatus('Stopped','00:02:00') }
Say "$s stopped"
}
function Sql-Init {
$envC = @((Get-ItemProperty $centralKey).Environment)
$cs = ($envC | Where-Object { $_ -like 'ScadaBridge__Database__ConfigurationDb=*' }) -replace '^[^=]*=',''
$script:sb = New-Object System.Data.SqlClient.SqlConnectionStringBuilder $cs
$script:db = $sb['Database']; if (-not $db) { $script:db = $sb.InitialCatalog }
if ($db -ne 'ScadaBridge') { throw "unexpected catalog" }
$env:SQLCMDPASSWORD = $sb.Password
}
function Sql-Scalar($q) { $r = & sqlcmd -S $sb.DataSource -d $db -U $sb.UserID -C -h -1 -W -b -Q "SET NOCOUNT ON; $q"; if ($LASTEXITCODE -ne 0) { throw "sql failed" }; ($r | Where-Object { $_ -and $_.Trim() } | Select-Object -First 1).Trim() }
$schemaQ = "SELECT CONCAT((SELECT COUNT(*) FROM __EFMigrationsHistory),'|',(SELECT MAX(MigrationId) FROM __EFMigrationsHistory),'|',(SELECT COUNT_BIG(*) FROM dbo.AuditLog))"
switch ($Action) {
'prep' {
Say "PREP $fromSha -> $sha"
if ((Get-FileHash "$stageDir\scadabridge-$sha.zip" -Algorithm SHA256).Hash -ne $zipSha256) { throw "zip hash mismatch" }
if (-not (Test-Path $namCentral)) { throw "missing $namCentral" }
Get-Content $namCentral -Raw | ConvertFrom-Json | Out-Null
if (Test-Path $stage) { Remove-Item $stage -Recurse -Force }
Expand-Archive "$stageDir\scadabridge-$sha.zip" $stage
$sv = (Get-Item "$stage\app\ZB.MOM.WW.ScadaBridge.Host.exe").VersionInfo.ProductVersion
$sc = (Get-Item "$stage\cli\scadabridge.exe").VersionInfo.ProductVersion
if ($sv -notlike "*+$sha*" -or $sc -notlike "*+$sha*") { throw "staged exe $sv / $sc" }
$lv = (Get-Item "$root\app\ZB.MOM.WW.ScadaBridge.Host.exe").VersionInfo.ProductVersion
if ($lv -notlike "*+$fromSha*") { throw "live exe $lv" }
Say "staged $sv; live $lv; app files $((Get-ChildItem "$stage\app" -Recurse -File).Count), cli $((Get-ChildItem "$stage\cli" -Recurse -File).Count)"
foreach ($s in 'ScadaBridge-Central','ScadaBridge-Site') {
& reg.exe export "HKLM\SYSTEM\CurrentControlSet\Services\$s" "$stageDir\_envbak-$s-$ts.reg" /y | Out-Null
if ($LASTEXITCODE -ne 0) { throw "reg export $s failed" }
$e = @((Get-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Services\$s").Environment)
Say ("$s env entries {0}" -f $e.Count); Digest $e | % { Say " env $s $_" }
}
$keep = New-Item -ItemType Directory -Force -Path "$root\_appsettings-keep-$ts"
foreach ($f in $settings) { Copy-Item "$root\app\$f" "$keep\$f"; Say ("keep {0} sha256 {1}" -f $f, (Get-FileHash "$keep\$f" -Algorithm SHA256).Hash) }
}
'backup' {
Sql-Init
$pre = Sql-Scalar $schemaQ; Say "schema before: $pre"
if (-not $pre.StartsWith('65|')) { throw "expected 65 migrations" }
& sqlcmd -S $sb.DataSource -d master -U $sb.UserID -C -b -Q "BACKUP DATABASE [$db] TO DISK = N'$bak' WITH COPY_ONLY, CHECKSUM, COMPRESSION, INIT" | Out-Null
if ($LASTEXITCODE -ne 0) { throw "backup failed" }
& sqlcmd -S $sb.DataSource -d master -U $sb.UserID -C -b -Q "RESTORE VERIFYONLY FROM DISK = N'$bak' WITH CHECKSUM" | Out-Null
if ($LASTEXITCODE -ne 0) { throw "verifyonly failed" }
Say "backup $bak VERIFYONLY ok"
}
'stop-site' { Stop-Svc 'ScadaBridge-Site' }
'stop-central' {
Stop-Svc 'ScadaBridge-Central'
$deadline = (Get-Date).AddSeconds(30)
while ((Get-Process -Name 'ZB.MOM.WW.ScadaBridge.Host' -ErrorAction SilentlyContinue) -and (Get-Date) -lt $deadline) { Start-Sleep 1 }
if (Get-Process -Name 'ZB.MOM.WW.ScadaBridge.Host' -ErrorAction SilentlyContinue) { throw "Host.exe still running" }
Say "no Host.exe remains"
}
'swap' {
if (Get-Process -Name 'ZB.MOM.WW.ScadaBridge.Host' -ErrorAction SilentlyContinue) { throw "Host.exe running" }
Rename-Item "$root\app" "app.bak.$ts"; Rename-Item "$root\cli" "cli.bak.$ts"
Say "renamed app/cli -> *.bak.$ts"
Copy-Item "$stage\app" "$root\app" -Recurse
Copy-Item "$stage\cli" "$root\cli" -Recurse
$keep = "$root\_appsettings-keep-$ts"
foreach ($f in $settings) {
Copy-Item "$keep\$f" "$root\app\$f" -Force
if ((Get-FileHash "$keep\$f" -Algorithm SHA256).Hash -ne (Get-FileHash "$root\app\$f" -Algorithm SHA256).Hash) { throw "appsettings mismatch $f" }
Say "restored $f (sha256 match)"
}
Copy-Item $namCentral "$root\app\appsettings.Central.json" -Force
Say ("installed NAM appsettings.Central.json sha256 {0}" -f (Get-FileHash "$root\app\appsettings.Central.json" -Algorithm SHA256).Hash)
Say ("live Host " + (Get-Item "$root\app\ZB.MOM.WW.ScadaBridge.Host.exe").VersionInfo.ProductVersion + " / CLI " + (Get-Item "$root\cli\scadabridge.exe").VersionInfo.ProductVersion)
$before = @((Get-ItemProperty $centralKey).Environment)
$keepEnv = @($before | Where-Object { -not (Is-Dropped $_) })
Set-ItemProperty -Path $centralKey -Name Environment -Value ([string[]]$keepEnv) -Type MultiString
$after = @((Get-ItemProperty $centralKey).Environment)
foreach ($e in $keepEnv) { if ($after -notcontains $e) { throw "env entry changed" } }
if (@($after | ? { Is-Dropped $_ }).Count -ne 0) { throw "GLAuth password entry still present" }
Say ("central env entries {0} -> {1}" -f $before.Count, $after.Count)
}
'start-central' { Say "starting ScadaBridge-Central"; Start-Service 'ScadaBridge-Central'; $ok = Wait-Ready 'http://localhost:8085/health/ready' 180; Say "central ready=$ok"; if (-not $ok) { throw "central not ready" } }
'start-site' { Say "starting ScadaBridge-Site"; Start-Service 'ScadaBridge-Site'; $ok = Wait-Ready 'http://localhost:8084/health/ready' 180; Say "site ready=$ok"; if (-not $ok) { throw "site not ready" } }
'verify' {
foreach ($s in 'ScadaBridge-Central','ScadaBridge-Site') { Say ("$s " + (Get-Service $s).Status) }
if ($env:COMPUTERNAME -eq 'WONDER-APP-VQ03') { Sql-Init; Say ("schema now: " + (Sql-Scalar $schemaQ)) }
}
'rollback-swap' {
foreach ($s in 'ScadaBridge-Site','ScadaBridge-Central') { Stop-Svc $s }
if (Test-Path "$root\app") { Rename-Item "$root\app" "app.failed.$ts" }
if (Test-Path "$root\cli") { Rename-Item "$root\cli" "cli.failed.$ts" }
Rename-Item "$root\app.bak.$ts" 'app'; Rename-Item "$root\cli.bak.$ts" 'cli'
& reg.exe import "$stageDir\_envbak-ScadaBridge-Central-$ts.reg" | Out-Null
Say "rolled back binaries + Central env (services left STOPPED; start Central vq03, vq04, then Sites)"
}
default { throw "bad action $Action" }
}
Remove-Item Env:SQLCMDPASSWORD -ErrorAction SilentlyContinue
Say "$Action OK"
exit 0
@@ -0,0 +1,57 @@
# 2026-11-19: DEV (wonder-app-vd03) `061432c8` → `2e0ce81e` (#144 + #145, no role no login), then NAM AD login on
**Owner go (2026-11-19):** "ok, do it. winrm usage approved (for the vd03 deploy)", as the first step of the DEV NAM
LDAP cutover (scadaproj plan `docs/plans/2026-11-15-nam-ldap-cutover.md` §0; record
`scadaproj/docs/deploy/2026-11-19-vd03-nam-ldap/`).
**Build:** `main` @ `2e0ce81e`, ProductVersion `1.0.0+2e0ce81ea04ac5e85ddb654cf33068594ba5c3c4`. Host and CLI
self-contained win-x64 from a clean detached worktree: 1056 app files, 224 cli files. The zip is 130 508 202 B, SHA-256
`6c706b0a76b7e9ec7ca00e12cea19c5d86814beaf66b541f527cbfc3d6572e23`.
**Delta:** 11 commits: #144 (`5a642d1c`, `2d8d5a3d`, `764e75c3`), #145 (`1d2170a1`, `a5243b1e`) and the #116
`leader-auto-down` default (`5bf6d1fa`, already drilled on vd03 2026-11-15). **No migration** (65 stays 65), no env change.
**Script:** `docs/deployment/deploy-prep/2026-11-19-144-145/dev-deploy.ps1`, derived from the 11-15 pack with the
migration and env steps removed. On-box notes: `E:\ApiInstall\ScadaBridge\_DEPLOY-NOTES-2e0ce81e-20261119T145229.txt`.
## Window
| Step | Local (EST) |
|---|---|
| Preflight: live `061432c8`, staged `2e0ce81e` Host+CLI, 65 migrations, AuditLog 2 856 rows | 14:52:30 → 14:52:42 |
| `reg export` both service keys; DB backup `E:\ApiInstall\ScadaBridge-backups\ScadaBridge_pre-2e0ce81e_20261119T145229.bak` (132 222 976 B, VERIFYONLY ok); appsettings kept | → 14:52:45 |
| Stop Site, then Central | 14:52:45 → 14:53:06 |
| Swap `app`/`cli` → `*.bak.20261119T145229`; appsettings restored (hash match) | → 14:53:12 |
| Central ready, Site ready | 14:53:24 / 14:53:33 |
**Downtime:** site data path ~47 s, central API ~30 s.
## Gates
- Health ready 200 (central `:8085`, site `:8084`); both services `FAILURE_ACTIONS_ON_NONCRASH_FAILURES: TRUE`.
- No retired-key warning in the central log.
- Bogus inbound `X-API-Key` → 401.
- CLI (login still off) `instance list` → 6 instances.
## Then: NAM AD login on (same day, 15:11)
- `appsettings.Central.json`: `ScadaBridge:Security:Ldap` → NAM (LDAPS nam1 + nam2 fallback, `svc_igntadmin`,
`${secret:nam-ldap-svc-password}`, `sAMAccountName`); the `Auth` block (`DisableLogin` + `AllowOutsideDevelopment`) removed.
- Central env: `ScadaBridge__Security__Ldap__ServiceAccountPassword` (GLAuth) removed, 12 → 11 entries.
- Central restarted; Site not restarted.
- **Zero-role proven live** before any `UG_WW_*` row existed: `dohertj2` was refused at `/auth/login`, at `/auth/token`
and on the CLI. Log: "the LDAP bind succeeded but none of the user's 176 group(s) maps to a ScadaBridge role".
- `LdapGroupMappings` rows 6–9 added by SQL: `UG_WW_APP_ADMIN_DEV`→Administrator, `UG_WW_APP_IDE_DEV`→Designer,
`UG_WW_APP_IDE_RO_DEV`→Viewer, `UG_WW_HIST_ADMIN_DEV`→Deployer (DEV interim until `UG_WW_APP_DEPLOY_DEV` exists).
- Then `dohertj2`: cookie login ✓, `/auth/token` 200 with Administrator+Designer+Deployer, CLI `instance list` and
`security role-mapping list` ✓.
- MxGateway connection Connected, 810/810, good 399 / bad 411 (baseline).
**Automation identity on DEV is now `dohertj2`** (HTTP Basic / `--password-file`); the login-off dev user is gone.
## Rollback
- **LDAP only:** restore `E:\ApiInstall\_ldapcut-20261119T145640\sb-appsettings.Central.json.pre-cutover` as
`app\appsettings.Central.json`, `reg import …\reg-ScadaBridge-Central.reg`, restart Central.
- **Binaries:** stop Site then Central; `app`/`cli` ← `*.bak.20261119T145229`; start Central then Site. No DB restore is
needed (schema unchanged); the backup above is there for safety.
@@ -0,0 +1,37 @@
# 2026-11-19: QA (wonder-app-vq03/vq04) `061432c8` → `2e0ce81e` (#144 + #145) + NAM AD login, one window
**Owner go (2026-11-19):** "do the qa ldap plan, winrm usage approved" — scadaproj plan
`docs/plans/2026-11-15-nam-ldap-cutover.md` §0Q; full record `scadaproj/docs/deploy/2026-11-19-qa-nam-ldap/`.
**Build:** the DEV artifact, zip SHA-256 `6c706b0a…6572e23` (record `2026-11-19-wonder-app-vd03-2e0ce81e.md`). No migration
(65 stays 65). Script: `docs/deployment/deploy-prep/2026-11-19-144-145/qa.tmpl.ps1` (the 11-15 QA template minus
`migrate`; its `swap` also installs each node's own NAM `appsettings.Central.json` and drops the GLAuth password env entry).
## Before the window
- `LdapGroupMappings` rows 6–10 added with the CLI as GLAuth `multi-role` (audited): `UG_WW_APP_ADMIN_QA`→Administrator,
`UG_WW_APP_IDE_QA`→Designer, `UG_WW_APP_IDE_RO_QA`→Viewer, `UG_WW_HIST_ADMIN_QA`→Deployer (interim),
`UG_WW_BTMGR_RO_QA`→Viewer.
- `nam-ldap-svc-password` seeded once into the shared SQL Server secret store (`zbsecrets`), with a Secrets CLI build that
honours `Secrets:Store:Provider=SqlServer` (the headless 0.6.6 CLI writes SQLite only). Both Centrals' KEK files match.
- DB backup `F:\MSSQL\BACKUP\ScadaBridge_pre-2e0ce81e_20261119T1600.bak` (VERIFYONLY ok).
## Window (EST)
Stop Site vq03 16:01:10, vq04 16:01:38; Central vq03 16:01:58, vq04 16:02:22. Swap both (central env 15 → 14). Central
vq03 ready 16:03:17, vq04 16:03:37; Site vq03 16:03:55, vq04 16:04:14. **Site data path ~2 min 17 s, central API ~55 s.**
## Gates
- `dohertj2`: cookie login ✓; `/auth/token` 200 with Administrator+Designer+Deployer; CLI (HTTP Basic) `instance list` (11)
and `security role-mapping list` ✓. Bogus user and GLAuth `multi-role` refused (UI and CLI).
- Active central vq03; site-a both nodes online; MxGateway Connected 2196/2196 Good; WarmCanary Connected 2/2 Good (also
after the OtOpcUa pair restart at 16:08–16:10); LocalDb replication connected.
- No LDAP / service-bind warning in the OTel log export.
**Automation identity on QA is now `dohertj2`.**
## Rollback
`rollback-swap` (binaries ← `app.bak.20261119T1600`, Central env ← `_envbak-ScadaBridge-Central-20261119T1600.reg`), then
start Central vq03, vq04, then the Sites. No DB restore needed. The `UG_WW_*` rows can stay.
@@ -0,0 +1,106 @@
# 2026-11-20: PROD cluster configuration (Site `site-par`, MxGateway API key, site MxGateway connection)
**Owner request:** "configure the clusters", with the option *Site + PSK + MxGateway conn*: create the Site entity, use
`SB-GRPC-PSK`, verify central↔site gRPC, mint an mxgw API key and add a site MxGateway data connection that points at
the vp03/vp04 gateways (as QA). No templates or instances.
Builds: ScadaBridge `2e0ce81e` (centrals wonder-app-vp03 10.220.153.205 / vp04 10.220.153.210; sites wonder-par-vp03
10.201.75.86 / par-vp04 10.201.75.87); MxAccessGateway `0.1.2+e74918cc` on vp03/vp04. Previous record:
[`2026-11-20-wonder-app-vp03-vp04-par-vp03-vp04-2e0ce81e.md`](2026-11-20-wonder-app-vp03-vp04-par-vp03-vp04-2e0ce81e.md).
Everything was driven from the Mac. ScadaBridge configuration went through the repo CLI (`2e0ce81e`-identical CLI
sources) over HTTP Basic (`SCADABRIDGE_AUTH_MODE=basic`, `--password-file`) against `http://wonder-app-vp03.zmr.zimmer.com:8085`
as `NAM\dohertj2`. WinRM was used only for mxgw deploy steps (key mint, store copy, service stop/start) and for reading logs.
No firewall changes. No secret was put on argv or written into this record.
## 1. §3A.5 checks on the active central (vp03)
| Check (UTC) | Result |
|---|---|
| 21:26:37 `GET /` | 302 → `/login`. `/health/active`: vp03 200 (active), vp04 503 (standby) |
| 21:26:55 `/auth/token` | **401**. Operator error: the password file had a trailing newline. The log shows only the 401; there is no LDAP or service-bind line |
| 21:27:16 `/auth/token` | **200**. `username=dohertj2`; roles **Administrator, Designer, Deployer**; claims `zb:username`, `zb:displayname`, `…/identity/claims/role` (×3), `iss`/`aud`/`nbf`/`exp`; `expires_in` 1500 s; `management_bearer=true` |
| 21:27:36 UI `/auth/login` | 302 → `/`, log `AuthenticationScheme: Cookies signed in`; `/` answers 200 with `dohertj2` and *Sign Out* in the page (cookie replayed as an explicit header: the 4-chunk ~10.8 KB cookie goes over curl's 8 KB cookie limit, the same probe note as in the mxgw PROD record) |
| 21:28:11 CLI over Basic | `site list` → `[]`, `data-connection list` → `[]`, rc 0 |
| Log | `E:\ApiInstall\ScadaBridge\central\logs\scadabridge-central-20261120.log` (FileStream shared read): no LDAP, `svc_igntadmin`, bind-failure or `system-side` lines, and no WRN/ERR from 16:28 to 16:36 local |
Not done: bogus-user refusal and zero-role refusal. We kept LDAP attempts to a minimum; the DEV/QA proofs (#144/#145) apply to the same build.
## 2. Site entity and central↔site gRPC
| Item | Value |
|---|---|
| Created 21:30:33 | Site **id 1**, Name `PAR`, SiteIdentifier `site-par`, Description "Wonderware PROD Site (PAR)" |
| Node A / B (Akka) | `akka.tcp://scadabridge@10.201.75.86:8082` / `akka.tcp://scadabridge@10.201.75.87:8082` |
| gRPC node A / B | `http://10.201.75.86:8083` / `http://10.201.75.87:8083` (IPv4 literals, matching QA's site-a shape) |
| PSK | central store secret `sb-grpc-psk-site-par` (seeded earlier); the site env carries the same value. No change was needed |
Evidence:
- Central log 21:30:34: `gRPC channel pair for site site-par refreshed (nodeA=true, nodeB=true)`. Site→central `CentralControlService/Heartbeat` calls on `:8083` return 200.
- `health site --identifier site-par` at 21:30:54: `isOnline=true`, firstSeen 21:30:20, nodeRole Active `10.201.75.86`, clusterNodes .86 Primary / .87 Standby both online, `localDbReplicationConnected=true`, oplog backlog 0, secret-mirror failures none on node-a and node-b.
- Site logs on both PAR boxes: the last "central endpoint unavailable" warning is from before the centrals started (par-vp03 21:25:35, par-vp04 21:25:10). It is followed by `preferred endpoint http://10.220.153.205:8083 is reachable again; site site-par failed back` (par-vp03 21:25:37, par-vp04 21:25:11). After that there are no refused or auth failures.
- Central→site round trip: `deploy artifacts --site-id 1` at 21:35:22. Deployment `1ba4d975a0b246a19c4e0f58f6ca6cc6` returned success. par-vp03 logged `SiteCommandService/ExecuteLifecycle` → `Deploying system artifacts` → `Applying 0 deployed secret(s)`. Site row `artifactsDeployedAt` = 21:35:22, hash `sha256:77d02460…`.
## 3. MxAccessGateway API key `scadabridge-prod`
QA reference (`scadabridge-qa`, read from an SMB copy of vq03's store; the copy was deleted afterwards): scopes
`events:read, invoke:read, invoke:write, metadata:read, session:close, session:open`, no constraints, no expiry,
display name = key id. QA shares one key across vq03/vq04 by copying the store, since both boxes have the same pepper. PROD does the same.
| Step (UTC) | Result |
|---|---|
| Pepper check | vp03 and vp04 `MxGateway__ApiKeyPepper` have the same sha256 prefix `d98f5191bba2`, which also matches Keychain `zb-prod/mxgw-pepper` |
| 21:31:19 mint on vp03 | `ZB.MOM.WW.MxGateway.Server.exe apikey create-key --key-id scadabridge-prod --display-name scadabridge-prod --scopes <the six above>`. It ran with the service's own env, exit 0, `Status: created`, created_utc 21:31:21. The token was written to an Administrators/SYSTEM-only file, pulled over SMB, stored in Keychain **`zb-prod/mxgw-apikey-scadabridge`** (account `scadabridge-prod`; read back and matched), and the remote file was deleted |
| 21:31:59 verify vp03 | `galaxy-test-connection` `{ok:true}`; `open-session` OK (first x86 Worker on PROD vp03, pid 20172) then `close-session` CLOSED; bogus key → `Missing or invalid API key` |
| 21:32:25–21:32:48 vp03 | `MxAccessGw` stopped (WAL checkpointed on close), `gateway-auth.db` pulled (sha256 `F71DFD15FD79…`), restarted, ready 200. About 23 s down, with no consumers yet |
| 21:33:09–21:34:07 vp04 | Stopped. Backup `E:\ApiInstall\MxAccessGateway\data\gateway-auth.db.bak-20261120-prod-keycopy` (vp04's own store, holding its earlier dashboard audit rows). Stale -wal/-shm removed, vp03's copy placed (hash verified), started. Ready **503**, as before: auth-store Healthy, worker-executable Healthy, galaxy-sql Unhealthy (scadaproj#60). Staging folder deleted |
| 21:34:16 verify vp04 | `open-session` OK (Worker pid 15684) and `close-session` CLOSED with the same token; bogus key → `Missing or invalid API key`. `galaxy-test-connection` returns `{}` here because of #60 |
The key store now holds only `scadabridge-prod`, and it is byte-identical on both boxes.
## 4. Site MxGateway data connection
QA reference (`data-connection list` on QA, read-only): connection 1 `MxGateway`, protocol `MxGateway`, retry 3, primary
`{"endpoint":"http://10.220.158.2:5120","apiKey":<literal, encrypted at rest>,"clientName":"","writeUserId":0,"useTls":false,"caFile":"","serverName":"","readTimeoutMs":5000}`,
no backup. QA's central secret store holds only `nam-ldap-svc-password`, so QA's apiKey is an **inline literal, not a `secret:` reference**.
PROD does the same and adds a backup endpoint at the owner's request.
| Item | Value |
|---|---|
| Created 21:35:14 | DataConnection **id 1**, site 1, Name `MxGateway`, Protocol `MxGateway`, FailoverRetryCount 3, `hasSecrets=true` |
| Primary | `http://10.220.153.205:5120` (vp03), apiKey = `scadabridge-prod` token (inline, encrypted at rest), useTls false, readTimeoutMs 5000 |
| Backup | `http://10.220.153.210:5120` (vp04), same token and settings |
| Delivery | Config files were 0600 in the session scratchpad (`--primary-config-file`/`--backup-config-file`) and deleted after use. Artifact deploy `1ba4d975…` at 21:35:22 |
Evidence: par-vp03 at 21:35:22 logged `Created DCL connection MxGateway (protocol=MxGateway)`. The site health report at
21:35:48 shows `dataConnectionStatuses {MxGateway: Connected}` and `dataConnectionEndpoints {MxGateway: "Connected to primary"}`.
This also proves the `10.201.75.x → 10.220.153.205:5120` path. **Tag quality cannot be checked**, because no instance binds the
connection yet (`dataConnectionTagQuality {}`). The backup (vp04) leg was proven only from the Mac (session open/close).
Its Galaxy browse needs #60.
## Left open
- Bogus-user and zero-role refusal on PROD (§3A.5 items 2 and 4): not run, to keep LDAP attempts low.
- vp04 Galaxy SQL access (scadaproj#60). Galaxy browse through vp04 fails until it is fixed. Session/read/write paths do not depend on it.
- When the PAR gateways exist (plan Q3), re-point connection 1 to `PAR-VP03:5120` / `PAR-VP04:5120` and redeploy artifacts. Mint the key on that pair then.
- Templates, instances, inbound API keys, bundle import: Phase 7, not in scope.
## Rollback
- Connection: `data-connection delete --id 1`, then `deploy artifacts --site-id 1`.
- Site: `site delete --id 1` (the site nodes go back to unregistered heartbeats).
- Key: `apikey revoke-key --key-id scadabridge-prod` on each box. On vp04 you can instead restore `gateway-auth.db.bak-20261120-prod-keycopy` with the service stopped.
## 2026-11-20 22:22Z: connection re-pointed to the PAR gateways (owner: "the par mxgw on the par scadabridge instances", chose "Re-point to PAR only")
- `data-connection update --id 1`: primary `http://10.201.75.86:5120` (par-vp03), backup `http://10.201.75.87:5120`
(par-vp04), same `scadabridge-prod` token and settings, retry 3. Config files 0600 in the scratchpad, deleted after use.
CLI built from a detached `2e0ce81e` worktree (assembly `scadabridge.dll`).
- `deploy artifacts --site-id 1` → deployment `2264d02f4b21450fa1ac61ad8161fb35`, success. par-vp03 logged
"Reconfigured DCL connection MxGateway ... in place" (no restart), then `Connect failed: Error connecting to subchannel`
retries. **Expected: mxgw is not installed on the PAR boxes** (held for WinPlatform, plan Q3). Site-par has no working
gateway until then; the connection retries (WRN log lines) and fails over between the two PAR endpoints.
- **When mxgw is installed on par-vp03/04:** use the same `MxGateway__ApiKeyPepper` (Keychain `zb-prod/mxgw-pepper`) and
copy vp03's `gateway-auth.db` (service stopped) so the `scadabridge-prod` token is valid there; then the connection
comes up on its own. Install script fix first: mxaccessgw#234.
- Rollback to the app-box gateways: the same `update` with `10.220.153.205:5120` / `10.220.153.210:5120`, then `deploy artifacts --site-id 1`.
@@ -0,0 +1,91 @@
# 2026-11-20: PROD site `site-app` co-located with the centrals on wonder-app-vp03 / vp04
**Owner request:** "prod should have a site too, configure it for mxaccessgw on central nodes". The owner chose
SiteIdentifier `site-app` and Name `APP`. The site follows the QA pattern: a site node runs next to the central on each
box and uses the mxgw gateway already running there.
Build: ScadaBridge `2e0ce81e` (`1.0.0+2e0ce81ea04ac5e85ddb654cf33068594ba5c3c4`). The site reuses the central's
`E:\ApiInstall\ScadaBridge\app`. Boxes: wonder-app-vp03 `10.220.153.205` (active central) and wonder-app-vp04
`10.220.153.210` (standby central). Previous record: [`2026-11-20-prod-cluster-config.md`](2026-11-20-prod-cluster-config.md).
Everything was driven from the Mac: `winrm` (NAM\adm_dohertj2) for the on-box deploy steps and log reads, and the
`2e0ce81e` CLI over HTTP Basic as `NAM\dohertj2` against `http://wonder-app-vp03.zmr.zimmer.com:8085` for configuration.
Nothing was done to `ScadaBridge-Central`, `MxAccessGw`, the PAR `ScadaBridge-Site` services, Site 1 or DataConnection 1.
No firewall changes. No secret went on argv or into this record.
## Ports per box
| Service | Akka | gRPC | HTTP |
|---|---|---|---|
| ScadaBridge-Central (unchanged) | 8081 | 8083 | 8085 (UI + health) |
| **ScadaBridge-Site (new, site-app)** | **8082** | **8086** | **8084** (health) |
| MxAccessGw / Overview / aaGR (unchanged) | | 5120 | 5130 / 5320 / 8090 |
Site paths: config `E:\ApiInstall\ScadaBridge\app\appsettings.Site.json` (the central loads only
`appsettings.Central.json`), and data and logs under `E:\ApiInstall\ScadaBridge\site\{data,logs}`. The site is a separate
Akka cluster from the centrals, with central memberCount still 2. LocalDb replication runs vp03 → `http://10.220.153.210:8086`.
## Steps (UTC)
| Time | Step | Result |
|---|---|---|
| 22:33:53 | `install-sb-site-colocated.ps1 -Mode dry` on vp03 | Aborted at the env step: `sb-site.env` was missing from `E:\psrem-stage\sb-2e0ce81e` on both boxes, because it had only been staged for PAR. The non-secret local file (sha256 `186210EE…`) was pushed to both boxes and the dry run was repeated |
| 22:34:26 | dry vp03 | `DONE mode=dry`. Host ProductVersion ok, KEK `FA4511BE`, ports 8082/8084/8086 free, 8 env entries |
| 22:34:35 | real vp03 | `service ScadaBridge-Site created STOPPED, env 8 entries`, `DONE mode=real`. Central Running, `/health/ready` 200, `/health/active` 200 |
| 22:34:43–22:34:46 | dry + real on vp04 | `DONE mode=dry`, then `DONE mode=real`. Central Running, ready 200, active 503 (standby, as before) |
| 22:35:04 | Seed on vp03: `seed-secret-prod.ps1 -Target ScadaBridge-Central -SecretName SB-GRPC-PSK-site-app -AllowRunning` | Shared SQL store `zbsecrets`, KEK `FA4511BE`, `set rc=0 {"name":"sb-grpc-psk-site-app","action":"set"} readback-match=True`, value file removed, `failures=0` |
| 22:35:20 | `site create` | **Site id 2**: Name `APP`, identifier `site-app`, description "PROD site co-located on the central nodes", Akka `akka.tcp://scadabridge@10.220.153.205:8082` / `@10.220.153.210:8082`, gRPC `http://10.220.153.205:8086` / `http://10.220.153.210:8086`. Central log at 22:35:20: `gRPC channel pair for site site-app refreshed (nodeA=true, nodeB=true)` |
| 22:35:25 | Started ScadaBridge-Site on vp03 and vp04 in parallel | Both Running by 22:35:28. vp03 founded the cluster (bootstrap guard "preferred founder") and vp04 was Up at 22:35:34 |
| 22:37:14 | **PROD-STAGE PATCH** (see below): vp04 `NodeName` changed to `node-b`, and vp04's site restarted | Started again by about 22:37:28. Bootstrap guard: "partner … REACHABLE … joining peer-first". `SiteReconciliationActor … node node-b` |
| 22:38:4x | Verification | See the evidence below |
| 22:38:55 | `data-connection create --site-id 2` | **DataConnection id 2**, site 2, `MxGateway` / protocol `MxGateway`, FailoverRetryCount 3, `hasSecrets=true`. Primary `http://10.220.153.205:5120`, backup `http://10.220.153.210:5120`, config shape as for site-par (`clientName ""`, `writeUserId 0`, `useTls false`, `readTimeoutMs 5000`). apiKey = the `scadabridge-prod` token (Keychain `zb-prod/mxgw-apikey-scadabridge`), inline and encrypted at rest. Config files were 0600 in the scratchpad and deleted after use |
| 22:39:00 | `deploy artifacts --site-id 2` | Deployment `63fdb88f49134bab89fa9892bb338fcb`, `site-app` success. vp03 site log: `Deploying system artifacts` → `Applying 0 deployed secret(s)` → `Created DCL connection MxGateway (protocol=MxGateway)` → `[MxGateway] Connection established` / `Entering Connected state` (22:39:04) |
| 22:40:26 / 22:40:58 | Overview registry on vp03, then vp04 | Old `appsettings.Prod.json` (sha `3CED9601…`) backed up as `.bak-20261120`. New file sha `964EE099…` placed in `app\` and in `E:\psrem-stage\overview-818775b3\`, then `Restart-Service ZbOverview`. Running, `/health/ready` 200 |
| 22:42 | Cleanup | `E:\psrem-stage\_values\sb-site-app` deleted on both boxes, and `_values\seed\site-app-psk.txt` is gone on vp03. Temporary log script removed |
## PROD-STAGE PATCH: vp04 NodeName
The staged `appsettings.Site.json.prod-vp04` had `"NodeName": "node-a"`, a copy of vp03's value. The PAR pair uses
`node-a` / `node-b`. NodeName feeds the audit `SourceNode`, the per-node `secret_mirror_status` row and the
reconciliation actor's identity, so two `node-a` nodes would overwrite each other's status. The fix changed the value to `node-b` and added a
`_comment_patch` key in three places: the local stage
(`scadaproj/docs/deploy/2026-11-19-prod-stage/sb/appsettings.Site.json.prod-vp04`), the vp04 staged copy and vp04's
installed `app\appsettings.Site.json` (sha `6B67CCFE…`; the earlier file is kept as `appsettings.Site.json.bak-20261120-nodename`).
Only vp04's new ScadaBridge-Site was restarted. vp04's staged `SHA256SUMS.txt` is now stale for that file and does not
list `sb-site.env`.
Afterwards the site health report lists `secretMirrorStatuses` for **node-a and node-b**, with no failures on either.
Cosmetic leftover: `_comment_Cluster` in both site-app configs still describes the PAR pair. It was not edited.
## Evidence
- `/health/ready` on :8084 returns 200 on both nodes: akka-cluster Up, memberCount 2, unreachable 0, leader `10.220.153.205:8082`,
roles `Site, site-site-app`. LocalDb shows `replicationConnected=true` and backlog 0 on both.
- `/health/active` on :8084 returns 200 on vp03 and 503 on vp04 ("Standby: active node … is …205:8082"), so exactly one node is active.
- vp03 logged `Replication session … faulted` warnings only while vp04 was restarting
(`Unavailable: This node is shutting down`). Replication reconnected after the restart.
- CLI `health site --identifier site-app`: `isOnline=true`, firstSeen 22:35:29, nodeRole Active `10.220.153.205`.
clusterNodes .205 Primary / .210 Standby, both online. `localDbReplicationConnected=true`, oplog backlog 0.
After the deploy (report 22:39:27): `dataConnectionEndpoints {MxGateway: "Connected to primary"}`.
`dataConnectionTagQuality {}` because no instance binds the connection yet.
- Site log: `Sitecentral transport: gRPC to 2 central endpoint(s) for site site-app`. No `Unauthenticated` or
`PermissionDenied` lines and no PSK errors.
- ScadaBridge-Central `/health/ready` returned 200 on both boxes at every check, with vp03 active (200) and vp04 standby (503) throughout.
The central log has no `svc_igntadmin`, bind-failure or LDAP-exception lines from this work. The only `[ERR]` is at
21:24:23Z (`akka-cluster … Removed`), which predates this work.
- site-par is unaffected. CLI `health site --identifier site-par` at 22:39:48 shows `isOnline=true`, Active `10.201.75.86`, and both nodes online.
- Overview on both centrals reports 10 instances: Up 7 / Down 1 (mxgw vp04, scadaproj#60) / Unreachable 2 (the PAR mxgw,
not installed). The ScadaBridge card has 3 clusters. `site-app` shows leader `…205:8082`, 2 members, node-a (vp03) Active Up,
and node-b (vp04) Standby Up. The group formerly named `site` is now `site-par` and is still Active/Standby Up. This is a
display/grouping label only.
## Rollback
1. On both boxes: `Stop-Service ScadaBridge-Site; sc.exe delete ScadaBridge-Site`. Then remove
`E:\ApiInstall\ScadaBridge\app\appsettings.Site.json` (plus `.bak-20261120-nodename` on vp04) and the
`E:\ApiInstall\ScadaBridge\site\` folder. Do not touch `app\` otherwise, because the central uses it.
2. CLI: `data-connection delete --id 2`, then `site delete --id 2`. Optionally delete the central-store secret
`sb-grpc-psk-site-app`.
3. Overview: restore `E:\ApiInstall\Overview\app\appsettings.Prod.json.bak-20261120` on both boxes (and the staged copy), then
`Restart-Service ZbOverview`. Revert the scadaproj stage file.
@@ -0,0 +1,141 @@
# 2026-11-20 — PROD first install: centrals wonder-app-vp03 + vp04 (installed, NOT started), site pair wonder-par-vp03 + vp04 `site-par` (installed + started), `2e0ce81e`
**Owner go (2026-11-20):** "use winrm to bring up ... scadabridge central. then scadabridge on the site servers." Owner decision
the same day: **the centrals are installed but not started** until the DBA has created the 104 missing nonclustered indexes in the
PROD config DB (scadaproj#59). The zbauth/zbsecrets startup migrators issue `CREATE INDEX`, and the app login `wwpdsbrw` has no
DDL rights. The sites are installed and started.
**Build:** `2e0ce81e`, ProductVersion `1.0.0+2e0ce81ea04ac5e85ddb654cf33068594ba5c3c4` (Host exe, read on all four boxes). Zip
`scadabridge-2e0ce81e.zip` SHA-256 `6C706B0A…6572E23`, checked by `install-sb.ps1` on every box. App 1056 files, cli 224.
**Transport:** `winrm` (deploy scope) running the staged `E:\psrem-stage\sb-2e0ce81e\install-sb.ps1` (dry run first, read, then
real) and `E:\psrem-stage\prod-common\seed-secret-prod.ps1` (SHA-256 prefix `3bd6237d0bd3ac00`, same as the scadaproj copy). The
KEK moved over SMB. No firewall changes. There were no security-software alerts and no unexpected WinRM errors.
**Not touched:** MxAccessGateway (`MxAccessGw`, `E:\ApiInstall\MxAccessGateway`, `mxgateway.kek`, `_values\mxgw-*`,
`nam-ldap-mxgw.txt`, vp04 `_values\seed`), which another session was installing at the same time. Nothing else on the boxes was
touched either.
## Timeline (UTC)
| Time | Step |
|---|---|
| 20:32:14 | vp03 Central dry run: clean (zip ok, config `appsettings.Central.json.prod-vp03` ok, 12 env entries); dry folder removed |
| 20:32:34–20:32:49 | vp03 Central real: expanded, **KEK generated**, service `ScadaBridge-Central` created STOPPED |
| ~20:33 | KEK pulled from vp03 over SMB (`C$\ProgramData\ZB\kek\scadabridge.kek`, 32 B) to a 0600 temp file, pushed to `E$\psrem-stage\_values\kek\` on vp04, par-vp03 and par-vp04, then the Mac copy was deleted. All three copies were 32 B with prefix `FA4511BE`. The folder ACL was inherited Administrators + SYSTEM FullControl only |
| 20:33:19 | vp04 Central dry run (`-KekFrom`): clean; dry folder removed |
| 20:33:37–20:33:51 | vp04 Central real: KEK copied from `-KekFrom`, service created STOPPED |
| 20:34:07 | vp03 seed `nam-ldap-svc-password` into the shared SQL store: **FAILED** (below) |
| 20:34:26 | par-vp03 Site dry run: clean (8 env entries; `SiteId site-par`, `NodeHostname 10.201.75.86`, seeds self-first, `PeerAddress http://10.201.75.87:8083`); dry folder removed |
| 20:34:44–20:34:55 | par-vp03 Site real: service `ScadaBridge-Site` created STOPPED |
| 20:34:57 | par-vp04 Site dry run: clean (self-first seeds `.87`, then `.86`; no `PeerAddress`, which is correct because only one node sets it); dry folder removed |
| 20:35:22–20:35:33 | par-vp04 Site real: service created STOPPED |
| 20:35:46 | `Start-Service ScadaBridge-Site` on both at once: par-vp04 Running at 20:35:48.71, par-vp03 at 20:35:48.96 |
| 20:36:02 / 20:36:27 | Site health checks (below) |
| 20:36:51 | Value cleanup (below) |
## Service configuration (env names only)
| | Central (vp03, vp04) | Site (par-vp03, par-vp04) |
|---|---|---|
| Service | `ScadaBridge-Central`, "ScadaBridge Central Node" | `ScadaBridge-Site`, "ScadaBridge Site Node" |
| Binary | `E:\ApiInstall\ScadaBridge\app\ZB.MOM.WW.ScadaBridge.Host.exe` | same |
| Account / start | LocalSystem / **Automatic** | LocalSystem / Automatic |
| Recovery | restart 5 s / 5 s / 10 s, reset 86 400, `failureflag 1` | same |
| Env (REG_MULTI_SZ) | 12: `SCADABRIDGE_CONFIG`, `ASPNETCORE_URLS`, `ScadaBridge__Database__ConfigurationDb`, `ScadaBridge__Security__JwtSigningKey`, `ScadaBridge__InboundApi__ApiKeyPepper`, `ScadaBridge__InboundApi__ApiKeyStore__SqlitePath`, `Secrets__SqlitePath`, `Secrets__MasterKey__Source`, `Secrets__MasterKey__FilePath`, `Secrets__SqlServer__ConnectionString`, `ScadaBridge__Telemetry__Exporter`, `ScadaBridge__Telemetry__OtlpEndpoint` | 8: `SCADABRIDGE_CONFIG`, `ScadaBridge__Communication__GrpcPsk`, `LocalDb__Replication__ApiKey`, `Secrets__SqlitePath`, `Secrets__MasterKey__Source`, `Secrets__MasterKey__FilePath`, `ScadaBridge__Telemetry__Exporter`, `ScadaBridge__Telemetry__OtlpEndpoint` |
| Role dir | `E:\ApiInstall\ScadaBridge\central\{data,logs}` | `E:\ApiInstall\ScadaBridge\site\{data,logs}` |
**KEK:** one `C:\ProgramData\ZB\kek\scadabridge.kek` for all four nodes, generated on vp03. SHA-256 prefix **`FA4511BE`** on
vp03, vp04, par-vp03 and par-vp04 (read back from the installed file). The `C:\ProgramData\ZB\kek` ACL has inheritance removed and
Administrators + SYSTEM FullControl only.
**Start type note:** the centrals are **Automatic**. A reboot of vp03/vp04 before scadaproj#59 closes would start them. They
would then fail on the zbauth/zbsecrets `CREATE INDEX`, and the failure actions would restart them 3 times. Set them to Manual or
Disabled if a reboot is possible before the DBA work.
## Seeds into the shared central store (vp03): FAILED, stopped as instructed
`seed-secret-prod.ps1 -Target ScadaBridge-Central -ValueFile E:\psrem-stage\_values\seed\nam-ldap-central.txt` (service
Stopped, SQL-aware CLI `E:\ApiInstall\_tools\secret-0.6.6-sql`):
```
value file read, length 15
value file removed: True
cmd.exe : secret: startup failed: Cannot find the object "zbsecrets.secret" because it does not exist or you do not
have permissions.
At E:\psrem-stage\prod-common\seed-secret-prod.ps1:46 char:10
+ CategoryInfo : NotSpecified: (secret: startup...ve permissions.:String) [], RemoteException
+ FullyQualifiedErrorId : NativeCommandError
```
- The CLI fails in its own startup. SQL Server returns this message (error 1088) for `CREATE INDEX`/`ALTER` on a table when
the login lacks the permission, so this is almost certainly the zbsecrets startup migrator's index DDL, which is the
scadaproj#59 gate. It is not a missing table: the 2026-11-20 schema check found `zbsecrets` present. No SQL workaround was
attempted.
- No secret was written. PowerShell 5.1 turned the CLI's stderr into a terminating `NativeCommandError` under
`$ErrorActionPreference='Stop'`, so the script never reached its readback/`SEED DONE` lines.
- **The script's `finally` block deleted `nam-ldap-central.txt` even though the seed failed.** It must be re-staged
(Keychain → 0600 → SMB) before the retry.
- **`SB-GRPC-PSK-site-par` was NOT attempted.** It would fail the same way and delete its value file. `site-psk.txt` (48 B, ACL
Administrators + SYSTEM) was **kept** in `E:\psrem-stage\_values\seed\` on vp03 for the retry.
- Script follow-up: the `finally` should delete the value file only on success. The CLI's stderr should be captured with
`$ErrorActionPreference='Continue'` around the call so the `rc`/readback line is printed.
## Site pair `site-par`: started, healthy
| | par-vp03 (10.201.75.86, node-a) | par-vp04 (10.201.75.87, node-b) |
|---|---|---|
| Service | Running | Running |
| `/health/ready` | **200** Healthy: akka-cluster Up, memberCount 2, unreachable 0, leader `.86`; localdb `replicationConnected=true`, backlog 0 | **200** Healthy: same; leader `.86`; `replicationConnected=true`, backlog 0 |
| `/health/active` | **200** active (oldest Up member for `site-site-par`) | **503** Standby, active = `.86` |
| `/health/live` | **404** (route not mapped on this build) | **404** |
| Roles | `Site`, `site-site-par` | same |
The two checks at 20:36:02 and 20:36:27 gave the same results.
**Logs** (`E:\ApiInstall\ScadaBridge\site\logs\scadabridge-site-20261120.log`):
- Akka: both started at 20:35:48–49; par-vp04 got `Welcome from [akka.tcp://scadabridge@10.201.75.86:8082]`; the leader `.86`
moved both nodes to Up (20:35:49.561 and 20:35:50.216).
- LocalDb: par-vp04 received `POST /localdb_sync.v1.LocalDbSync/Sync` from its peer at 20:35:49.430; health shows the stream
connected on both sides.
- Central gRPC: expected failures, e.g. `Central control-plane endpoint http://10.220.153.205:8083 is unavailable; site site-par
failed over to http://10.220.153.210:8083`, then `Status(StatusCode="Unavailable" ... SocketException (10061): No connection
could be made because the target machine actively refused it.)`, and `Reconcile pass 1/3 ... did not complete`. On par-vp03
(the active node) the only `[ERR]` was `Failed to send health report for site site-par` with the same 10061. **10061 means
"refused", not a timeout.** The 10.201.75.x → 10.220.153.x path to `:8083` is open and the centrals are simply not
listening.
- Other warnings are all routine startup ones: `ClusterSingletonProxy failed to find ... [deployment-manager]/[event-log-handler]`
before Up, and `HTTP/2 is not enabled for [::]:8084` (no TLS). There were no other WRN lines and no other ERR/FTL lines
(par-vp03: 1 ERR as above; par-vp04: 0).
The sites were not restarted.
## Cleanup (20:36:51)
| Box | Removed | Left |
|---|---|---|
| vp03 | `_values\sb-central` | `_values\seed\site-psk.txt` (kept for the PSK seed); `nam-ldap-central.txt` was already deleted by the seed script; `_values\mxgw-vp03` not touched |
| vp04 | `_values\sb-central`, `_values\kek` | `_values\mxgw-vp04` and `_values\seed` not touched (not this install's) |
| par-vp03 | `_values\sb-site`, `_values\kek` | — |
| par-vp04 | `_values\sb-site`, `_values\kek` | — |
The service env in the registry now holds the only on-box copy of the central/site values.
## State and next steps
**Centrals installed, NOT started — gated on scadaproj#59 (DBA indexes).** Then:
1. Re-stage `nam-ldap-central.txt` on vp03 and run both seeds on vp03 (`nam-ldap-svc-password`, then
`-SecretName SB-GRPC-PSK-site-par -Description 'ScadaBridge site-par gRPC PSK (PROD)'` from the kept `site-psk.txt`).
Each must show readback-match=True and failures=0.
2. Start vp03, wait for `/health/ready` 200, then start vp04.
3. Run the §3A.5 checks with `dohertj2` on the active central.
4. Create the Site entity `site-par` (`GrpcNodeA/BAddress` `http://<PAR-VP0x>:8083`).
5. Mint the mxgw API key and create the MxGateway data connection.
## Parent follow-up (2026-11-20)
- `ScadaBridge-Central` start type set to **Manual** on vp03 and vp04 (`sc.exe config … start= demand`), so a reboot
before scadaproj#59 closes cannot crash-loop the service. Set it back to Automatic (`start= auto`) at first start.
- `seed-secret-prod.ps1` fixed on all four boxes: it now keeps the value file when a seed fails, so a retry needs no
re-staging. `nam-ldap-central.txt` must still be re-staged from the Keychain (`btdb-manager/ldap`) before the retry,
because the old copy deleted it; `site-psk.txt` is still on vp03.
@@ -0,0 +1,43 @@
# 2026-11-22: PROD CVD reactor content (templates, BTDB, inbound API methods, Z28061–Z28069 on site-par, API keys)
**Owner requests (2026-11-22):** first "start the plan but wait to deploy to the par sites" (configuration only),
then "complete the plan. copy the mes and delmia api keys to a text file locally so i can send to the teams".
Plan: scadaproj `docs/plans/2026-11-22-prod-cvd-content-copy.md` §4 steps 0–9. **Step 10 (cutover of MES/Delmia
callers, QA instances) not done**: that is for the owner and the teams.
Build: ScadaBridge `2e0ce81e` (centrals wonder-app-vp03 active / vp04 standby; site `site-par` id 1 on par-vp03/04).
Everything was driven from the Mac through the repo CLI over HTTP Basic (`--password-file`) against
`http://wonder-app-vp03.zmr.zimmer.com:8085` as `NAM\dohertj2`, plus read-only SQL (PROD BT as `wonderapp`) and the
read-only mxgw client (galaxy-browse; `smoke --item` reads without `--value`). No WinRM, no firewall changes, no secret on
argv or in this record. QA was read only. site-app (id 2) and data connection 2 were not touched.
## Results
| Step | Result |
|---|---|
| 0 BT re-check | PROD BT (WONDER-SQL-VP03/BT) section 1: 9 machines + group `CVD_Reactors` (id 20) "exists", 0 conflicts. No insert |
| 1 Mapping | Re-browsed CVDAisle_1 (6762): 108 objects, 0 differences from the 2026-11-22 snapshot; all 9×244 binding refs and alarm prefixes belong to the right machine |
| 2 Templates | Bundle `cvdreactor-templates.scadabundle` (sha256 `6dcdc453…235e3`): preview 11 adds / 0 blockers (20 advisory scanner references); import `--on-conflict skip` → 11 added (import `98b04de4-96fd-4966-8dff-127d55fdc4c8`). Ids 1–11, **CvdReactor = 5**. `template validate --id 5` reports 3 alarm-attribute errors (`Left/RightReactorSide.MTA_Lockout`, `RightReactorSide.MantleTemp` "not in flattened configuration"); QA template 1 reports the same 3, and the deploy gate accepted all 9 instances |
| 3 BTDB | db-connection **id 1 `BTDB`** (`wondersqlpd`, `wonderapp`, `TrustServerCertificate=True;MultiSubnetFailover=True`, string via `--connection-string-file`). `deploy artifacts --site-id 1` → success (deployment `e48d8cfa…`); site-par artifacts 16:37:47Z `sha256:f46e0c04…`; site-app unchanged (12:43:14Z) |
| 4 API methods | 7 created from `qa-export/api-methods.json`, identical to QA: 1 MesMoveIn (46 s), 2 MesMoveOut, 3 DelmiaRecipeDownload, 4 SimpleAlarmStatusRequest, 5 MesSampleData, 6 MesIsMoveInReady, 7 MesIsMoveOutReady. `IpsenMESMoveIn` not created |
| 5 Area | **`CVDAisle_1` id 1** on site-par |
| 6–7 Instances | Z28063=1, Z28061=2, Z28062=3, Z28064=4, Z28065=5, Z28066=6, Z28067=7, Z28068=8, Z28069=9; each 244 bindings on connection 1 + 7 native alarm source overrides (equal to the generated files). `deploy instance` (which also enables) → **9 × Success**, all Enabled. Snapshot ~45 s after deploy: **244/244 Good on every instance** (root 16, Left/RightReactorSide 60, LeakTest 23 each, Left/RightMESReceiver 26, DelmiaReceiver 10). Z28063 spot-checked against the gateway: `MachineCode` '3', `Left_003.CycleRunning` False, `Left_003.LeakTest.LeakTestActive` False, MoveInReadyFlag L False / R True: all equal. 10 alarm states (3 computed inactive, 7 native placeholders: the mxgw alarm area is still off) |
| 8 API keys | `MES_Key` (keyId `4dd9f6f6…`) = DelmiaRecipeDownload, MesIsMoveInReady, MesIsMoveOutReady, MesMoveIn, MesMoveOut, MesSampleData, SimpleAlarmStatusRequest (QA's set minus IpsenMESMoveIn); `delmia-notifier` (keyId `b6dd27c5…`) = DelmiaRecipeDownload. Tokens in the Mac Keychain `zb-prod/sb-inbound-MES_Key`, `zb-prod/sb-inbound-delmia-notifier` and in two 0600 handover files on the owner's Desktop. A first create pair (`1ace3a97…`, `6ea4ae48…`) was deleted unused because its tokens were not captured |
| 9 Smoke | `MesIsMoveInReady {}` → 400 VALIDATION_FAILED. `103564_A` → 200 false, `_B` → 200 true (= snapshot). `<SAPID>_A` for each of the other 8 → 200, each equal to that instance's Left MoveInReadyFlag. `SimpleAlarmStatusRequest 103564` → 200 `{"WasSuccessful":true,"Alarms":[]}`. `DelmiaRecipeDownload {}` with either key → 200 `{"Result":false,"ResultText":"MachineCode is required"}` (no required fields; returns before routing). delmia-notifier → MesIsMoveInReady → **403 NOT_APPROVED**; no key → 401. **Standby vp04 also serves `/api`** (400 probe, and a real read-only MesIsMoveInReady 200) |
## Findings
- `MesIsMoveInReady` with a SAPID with no leading digits (`"_A"`) returns **500** "Method return value did not match its
return definition" instead of a clean `WasSuccessful=false` (8 such calls were sent by a shell-quoting mistake; the method is read-only).
- DelmiaRecipeDownload routes `Route.To(MachineCode)` with the value as sent: the caller must send the plain machine code
(`Z28063`), not `Z28063A`.
- QA template `RightMESReceiver` (QA id 6) lacks 14 inherited attribute rows and the 2 inherited scripts that PROD id 9
and both `LeftMESReceiver`s carry; the effective configuration is the same.
- Native alarm mirroring stays empty until the mxgw alarm area is set (plan C4).
Run log (scratch, not kept): `run/run-log.md` in the session scratchpad.
## Rollback (before cutover)
`instance disable`/`delete` ×9, `api-method delete` ×7, `security api-key delete` ×2, `template delete` (derived before
bases), `db-connection delete --id 1`, `deploy artifacts --site-id 1`. QA keeps serving MES throughout.
+1 -1
View File
@@ -547,7 +547,7 @@ file/stdin forms for any value that carries a key or password.
- **Connection failure**: If the CLI cannot connect to the management URL (e.g., DNS failure, connection refused), it exits with code 1 and a descriptive error message.
- **Command timeout**: Ordinary CLI commands use a 30-second client deadline. All instance/artifact/site deployment aliases defer the overall deadline to the server, retaining a bounded socket-connect timeout. Server Ask expiry returns HTTP 504; a client deadline is reported locally. Bundle and Flow Debugger commands retain their explicit timeout policies.
- **Authentication failure**: If the server returns HTTP 401 (LDAP bind failed at `/auth/token` or on a Basic call, or a bearer token that a fresh exchange could not replace), the CLI exits with code 1.
- **Authentication failure**: If the server returns HTTP 401 (LDAP bind failed at `/auth/token` or on a Basic call, or a bearer token that a fresh exchange could not replace), the CLI exits with code 1. A user whose directory groups map to no ScadaBridge role gets the same `401` "Invalid username or password." as a wrong password, from `/auth/token` and on a Basic call alike — the server does not reveal which it was (ScadaBridge#144/#145; `Component-Security.md` *No role, no login*). Check the user's group mappings when valid credentials are refused.
- **Authorization failure**: If the server returns HTTP 403, the CLI exits with code 2.
## Dependencies
@@ -65,7 +65,7 @@ Content-Type: application/json
- `ManagementSuccess` → HTTP 200 with JSON body
- `ManagementError` → HTTP 400 with `{ "error": "...", "code": "..." }`
- `ManagementUnauthorized` → HTTP 403 with `{ "error": "...", "code": "UNAUTHORIZED" }`
- Authentication failure → HTTP 401
- Authentication failure → HTTP 401 (including a caller whose groups map to no role, or a bearer token with no role claim — `Component-Security.md` *No role, no login*)
- Actor timeout → HTTP 504
- Large request while another is in flight → HTTP 409 with `{ "error": "...", "code": "IMPORT_IN_PROGRESS" }` (see below)
+6 -3
View File
@@ -32,7 +32,7 @@ Central cluster. Sites do not have user-facing interfaces and do not perform ind
Set in a local or docker-dev environment via the environment variable `ScadaBridge__Security__Auth__DisableLogin=true`. Note that `ScadaBridge:Security:Auth` is a child sub-section nested inside `ScadaBridge:Security`.
## No role, no login (ScadaBridge#144)
## No role, no login (ScadaBridge#144, #145)
A successful LDAP bind is **not** enough to sign in: the user's groups must map to **at least one** ScadaBridge role. This is the family standard (scadaproj NAM AD cutover plan §3.2), and it matters once the LDAP search base is domain-wide — without it, any directory user could sign in role-less and read the bare-`[Authorize]` pages (`/` and `/monitoring/health`).
@@ -40,7 +40,10 @@ A successful LDAP bind is **not** enough to sign in: the user's groups must map
- **Indistinguishable from a bad password.** The client gets exactly the bad-credentials response — `/auth/login` redirects to `/login?error=Invalid username or password.` with no session cookie; `/auth/token` answers `401` `{ error: "Invalid username or password.", code: "AUTH_FAILED" }` with no token. The refusal also **counts as a login-throttle failure**, so the throttle cannot be used to tell "right password, no role" from "wrong password".
- **Logged server-side only.** Each refusal is logged at Warning (category `ZB.MOM.WW.ScadaBridge.CentralUI.Auth.AuthEndpoints`) with the endpoint, the username and the reason (no mapped role, with the group count; or the mapping exception). There is no login audit event in the Audit Log — sign-in success and failure are not audited today — so the log line is the record.
- **Sessions end the same way.** A cookie session whose groups map to zero roles on a refresh is ended (see *Cookie session lifecycle*).
- **HTTP Basic is not covered yet.** `ManagementAuthenticator`'s Basic branch (`POST /management`, the audit REST endpoints, the debug-stream hub) still authenticates a zero-role bind. Role-gated commands deny it, but the read-only management queries that `ManagementActor.GetRequiredRoles` opens to *any authenticated user* answer it. Closing that is a follow-up to ScadaBridge#144.
- **The management/CLI surfaces refuse too (ScadaBridge#145).** `ManagementAuthenticator` is the only source of the caller's identity for `POST /management`, the audit REST endpoints (`/api/audit/*`) and the debug-stream hub (`/hubs/debug-stream`), so the rule is enforced once there and covers all three — including the read-only commands `ManagementActor.GetRequiredRoles` opens to *any authenticated user*, which a zero-role caller would otherwise reach.
- **HTTP Basic:** a bind whose groups map to zero roles, or whose role mapping throws, answers exactly what a bad password answers on that surface — `401` `{ error: "Invalid username or password.", code: "AUTH_FAILED" }` (the hub aborts the connection) — and counts as a login-throttle failure; a refusal no longer clears earlier failures the way a successful bind does.
- **Bearer:** a token with no role claim is refused with the invalid-token response, `401` `{ error: "Invalid or expired bearer token.", code: "AUTH_FAILED" }`. `/auth/token` no longer mints such a token, but one minted before the rule shipped stays signed and unexpired for up to `JwtExpiryMinutes`. Like every bearer failure it involves no LDAP bind, so it does not feed the login throttle. A bearer request performs no role mapping, so a mapper fault does not affect it.
- Each refusal is logged at Warning (category `ZB.MOM.WW.ScadaBridge.ManagementService.ManagementAuthenticator`) with the endpoint (request path), the username and the reason.
- Under the dev/test `DisableLogin` flag none of this runs: the auto-login identity holds every role.
## Session Management
@@ -55,7 +58,7 @@ A successful LDAP bind is **not** enough to sign in: the user's groups must map
- `POST /auth/token` (form `username`/`password`) performs the LDAP bind (login-throttled), maps the groups to roles and site scope (zero roles, or a mapping fault, is refused as a bad password — *No role, no login* above), and returns a short-lived **HS256 JWT** signed with `SecurityOptions.JwtSigningKey`, shared by both central nodes: `{ access_token, token_type: "Bearer", expires_in, management_bearer: true, username, display_name, roles }`.
- **JWT claims**: display name, username, roles, and for a site-scoped Deployer the permitted site IDs (no site claims = system-wide, the same normalisation the Basic path uses). Issuer and audience are both fixed to `scadabridge-central`. Lifetime is `JwtExpiryMinutes` (default **15**).
- `ManagementAuthenticator` accepts `Authorization: Bearer <jwt>` alongside HTTP Basic on every CLI-facing surface: `POST /management`, the audit REST endpoints, and the debug-stream hub (`/hubs/debug-stream`, token in the `Authorization` header of the negotiate request and WebSocket upgrade). Validation checks the HS256 signature, issuer, audience and lifetime (30 s clock skew for node-to-node drift; `nbf` is back-dated by the same skew so a peer whose clock lags the issuer accepts a fresh token after failover); unsigned (`alg=none`) or other-algorithm tokens are rejected, and a failure is a plain `401 AUTH_FAILED` (no LDAP bind, so it does not feed the login throttle). The token is never logged.
- **Roles and scope inside a valid token are authoritative for its lifetime**: a bearer request performs no LDAP bind and no role re-map. A role revocation therefore reaches a bearer caller when its token expires (≤ `JwtExpiryMinutes`); rotating `JwtSigningKey` invalidates every outstanding token immediately.
- **Roles and scope inside a valid token are authoritative for its lifetime**: a bearer request performs no LDAP bind and no role re-map. A token that carries no role claim is refused (*No role, no login* above). A role revocation therefore reaches a bearer caller when its token expires (≤ `JwtExpiryMinutes`); rotating `JwtSigningKey` invalidates every outstanding token immediately.
- The CLI exchanges credentials once per process, keeps the token in memory only, re-exchanges once on a `401`, and falls back to HTTP Basic when the server has no bearer support — see `Component-CLI.md`. HTTP Basic stays supported on every surface.
- Under the dev/test `DisableLogin` flag `/auth/token` answers `404`: the management surfaces already authenticate every request as the dev user, and no credential-free token is minted that another deployment sharing the key might honour.
- **Signing key**: `StartupValidator` refuses to boot a Central node whose `JwtSigningKey` is missing or shorter than **32 UTF-8 bytes** (256 bits, the HMAC-SHA256 floor). `JwtTokenService` repeats the length check at construction as defence in depth.
@@ -1,6 +1,8 @@
using System.Text;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.Auth.Abstractions.Ldap;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Management;
@@ -42,9 +44,24 @@ namespace ZB.MOM.WW.ScadaBridge.ManagementService;
/// <see cref="AuthenticatedUser.PermittedSiteIds"/>). The startup <c>AUTH DISABLED</c> warning
/// (emitted by <c>AddSecurity</c>) already announces this posture. Dev/test ONLY.
/// </para>
/// <para>
/// <b>No role, no login</b> (ScadaBridge#145, the management-surface half of #144; scadaproj NAM
/// cutover plan §3.2): a Basic bind whose groups map to zero roles, or whose role mapping throws,
/// is refused exactly like a bad password — the same <c>401 AUTH_FAILED</c> body and a
/// <see cref="LoginThrottle"/> failure — and a bearer token that carries no role claim is refused
/// like any other invalid token. Every refusal is logged at Warning under
/// <see cref="LoggerCategory"/> with the endpoint, the username and the reason. Because the three
/// callers (<see cref="ManagementEndpoints"/>, <see cref="AuditEndpoints"/>, the debug-stream hub)
/// take their <see cref="AuthenticatedUser"/> only from here, none of them can admit a role-less
/// principal — including the commands <c>ManagementActor.GetRequiredRoles</c> opens to any
/// authenticated user.
/// </para>
/// </remarks>
public static class ManagementAuthenticator
{
/// <summary>The logger category for the sign-in refusals logged by this authenticator.</summary>
public const string LoggerCategory = "ZB.MOM.WW.ScadaBridge.ManagementService.ManagementAuthenticator";
/// <summary>Outcome of <see cref="AuthenticateAsync"/>: exactly one of the two fields is set.</summary>
/// <param name="User">The authenticated principal on success; otherwise <see langword="null"/>.</param>
/// <param name="Failure">The 401 <see cref="IResult"/> to return on failure; otherwise <see langword="null"/>.</param>
@@ -143,11 +160,19 @@ public static class ManagementAuthenticator
new { error = LdapAuthFailureMessages.ToMessage(authResult.Failure), code = "AUTH_FAILED" }, statusCode: 401));
}
throttle.RecordSuccess(username, remoteIp!);
// 3. Role resolution. No role, no login (ScadaBridge#145, mirrors #144's /auth/login and
// /auth/token): a successful bind whose groups map to zero roles — or whose mapping
// throws — is refused exactly like a bad password: same 401 body, a throttle failure.
// The reason is logged server-side only.
var mappingResult = await ResolveBasicRolesAsync(context, authResult, username);
if (mappingResult is null)
{
throttle.RecordFailure(username, remoteIp!);
return new AuthOutcome(null, Results.Json(
new { error = LdapAuthFailureMessages.InvalidCredentials, code = "AUTH_FAILED" }, statusCode: 401));
}
// 3. Role resolution
var roleMapper = context.RequestServices.GetRequiredService<RoleMapper>();
var mappingResult = await roleMapper.MapGroupsToRolesAsync(authResult.Groups, context.RequestAborted);
throttle.RecordSuccess(username, remoteIp!);
var permittedSiteIds = mappingResult.IsSystemWideDeployment
? Array.Empty<string>()
@@ -162,10 +187,67 @@ public static class ManagementAuthenticator
return new AuthOutcome(resolvedUser, null);
}
/// <summary>
/// Maps a successful Basic bind's groups to roles, enforcing <b>no role, no login</b>
/// (ScadaBridge#145). Returns <see langword="null"/> — the caller then answers exactly as for a
/// bad password — when the groups map to zero roles or the mapper throws; the refusal is logged
/// at Warning with the endpoint, the username and the reason. Cancellation of the request is not
/// a refusal and propagates.
/// </summary>
/// <param name="context">The current HTTP context (services, request path, request-aborted token).</param>
/// <param name="authResult">The successful LDAP bind result.</param>
/// <param name="username">The username as sent, used when the bind returned no canonical name.</param>
/// <returns>The role mapping, or <see langword="null"/> when the request must be refused.</returns>
private static async Task<RoleMappingResult?> ResolveBasicRolesAsync(
HttpContext context, LdapAuthResult authResult, string username)
{
var logger = Logger(context);
var endpoint = EndpointName(context);
var who = string.IsNullOrEmpty(authResult.Username) ? username : authResult.Username;
RoleMappingResult mappingResult;
try
{
var roleMapper = context.RequestServices.GetRequiredService<RoleMapper>();
mappingResult = await roleMapper.MapGroupsToRolesAsync(authResult.Groups, context.RequestAborted);
}
catch (OperationCanceledException) when (context.RequestAborted.IsCancellationRequested)
{
throw;
}
catch (Exception ex)
{
logger.LogWarning(ex,
"Sign-in refused at {Endpoint} for {Username}: the LDAP bind succeeded but role mapping failed. " +
"The client sees the bad-credentials response.",
endpoint, who);
return null;
}
if (mappingResult.Roles is not { Count: > 0 })
{
logger.LogWarning(
"Sign-in refused at {Endpoint} for {Username}: the LDAP bind succeeded but none of the user's {GroupCount} " +
"group(s) maps to a ScadaBridge role. The client sees the bad-credentials response.",
endpoint, who, authResult.Groups.Count);
return null;
}
return mappingResult;
}
private static ILogger Logger(HttpContext context) =>
context.RequestServices.GetService<ILoggerFactory>()?.CreateLogger(LoggerCategory) ?? NullLogger.Instance;
private static string EndpointName(HttpContext context) =>
context.Request.Path.HasValue ? context.Request.Path.Value! : "management";
/// <summary>
/// Validates a bearer JWT and projects its claims onto an <see cref="AuthenticatedUser"/>.
/// The token's roles and site scope are taken as-is (authoritative until expiry); an
/// empty site-id set is the same system-wide signal the Basic path produces.
/// empty site-id set is the same system-wide signal the Basic path produces. A token with
/// no role claim is refused like an invalid token (ScadaBridge#145): <c>/auth/token</c> no
/// longer mints one, but a token minted before that rule shipped stays valid until it expires.
/// </summary>
private static AuthOutcome AuthenticateBearer(HttpContext context, string token)
{
@@ -181,8 +263,21 @@ public static class ManagementAuthenticator
var displayName = principal.FindFirst(JwtTokenService.DisplayNameClaimType)?.Value;
var roles = principal.FindAll(JwtTokenService.RoleClaimType)
.Select(c => c.Value)
.Where(r => !string.IsNullOrWhiteSpace(r))
.Distinct(StringComparer.OrdinalIgnoreCase)
.ToArray();
// No role, no login (ScadaBridge#145): the same 401 as an invalid token. Like any other
// bearer failure it performs no LDAP bind, so it does not feed the login throttle.
if (roles.Length == 0)
{
Logger(context).LogWarning(
"Sign-in refused at {Endpoint} for {Username}: the bearer token carries no ScadaBridge role. " +
"The client sees the invalid-token response.",
EndpointName(context), username);
return new AuthOutcome(null, Results.Json(
new { error = "Invalid or expired bearer token.", code = "AUTH_FAILED" }, statusCode: 401));
}
var permittedSiteIds = principal.FindAll(JwtTokenService.SiteIdClaimType)
.Select(c => c.Value)
.Distinct(StringComparer.Ordinal)
@@ -0,0 +1,37 @@
using System.Collections.Concurrent;
using Microsoft.Extensions.Logging;
namespace ZB.MOM.WW.ScadaBridge.ManagementService.Tests;
/// <summary>An in-memory logger provider that records every formatted log line with its category and level.</summary>
internal sealed class CapturingLoggerProvider : ILoggerProvider
{
/// <summary>One captured log entry.</summary>
/// <param name="Category">The logger category.</param>
/// <param name="Level">The log level.</param>
/// <param name="Message">The formatted message.</param>
/// <param name="Exception">The logged exception, if any.</param>
public sealed record Entry(string Category, LogLevel Level, string Message, Exception? Exception);
/// <summary>Every entry logged through this provider, in order.</summary>
public ConcurrentQueue<Entry> Entries { get; } = new();
/// <inheritdoc />
public ILogger CreateLogger(string categoryName) => new Logger(categoryName, Entries);
/// <inheritdoc />
public void Dispose()
{
}
private sealed class Logger(string category, ConcurrentQueue<Entry> sink) : ILogger
{
public IDisposable? BeginScope<TState>(TState state) where TState : notnull => null;
public bool IsEnabled(LogLevel logLevel) => true;
public void Log<TState>(LogLevel logLevel, EventId eventId, TState state, Exception? exception,
Func<TState, Exception?, string> formatter) =>
sink.Enqueue(new Entry(category, logLevel, formatter(state, exception), exception));
}
}
@@ -0,0 +1,531 @@
using System.Net;
using System.Runtime.CompilerServices;
using System.Text;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Http.Connections.Features;
using Microsoft.AspNetCore.Http.Features;
using Microsoft.AspNetCore.SignalR;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using NSubstitute;
using NSubstitute.ExceptionExtensions;
using ZB.MOM.WW.Audit;
using ZB.MOM.WW.Auth.Abstractions.Ldap;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Management;
using ZB.MOM.WW.ScadaBridge.Communication;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
using ZB.MOM.WW.ScadaBridge.Security;
using ZB.MOM.WW.ScadaBridge.Security.Auth;
namespace ZB.MOM.WW.ScadaBridge.ManagementService.Tests;
/// <summary>
/// ScadaBridge#145 — the management/CLI surfaces (<c>POST /management</c>, the audit REST endpoints and
/// the debug-stream hub) refuse a principal with no ScadaBridge role, mirroring #144's rule for
/// <c>/auth/login</c> and <c>/auth/token</c>. Over HTTP Basic, a bind whose groups map to zero roles, or
/// whose role mapping throws, is refused exactly like a bad password (same 401 body, a login-throttle
/// failure); over Bearer, a token with no role claim is refused like an invalid token. Each refusal is
/// logged at Warning with the endpoint, the username and the reason.
/// </summary>
public class ManagementZeroRoleRefusalTests
{
private const string Category = ManagementAuthenticator.LoggerCategory;
private const string Key = "this-is-a-test-signing-key-for-hmac-sha256-must-be-long-enough";
private const string Ip = "10.0.0.7";
/// <summary>The role outcome the substituted <see cref="RoleMapper"/> produces.</summary>
public enum Mapping
{
/// <summary>The groups map to no role.</summary>
ZeroRoles,
/// <summary>The mapper throws (e.g. the configuration database is unreachable).</summary>
Throws,
/// <summary>The groups map to exactly one role (the one the surface needs).</summary>
OneRole,
}
private sealed class Rig : IDisposable
{
public required ServiceProvider Provider { get; init; }
public required LoginThrottle Throttle { get; init; }
public required CapturingLoggerProvider Logs { get; init; }
public required RoleMapper RoleMapper { get; init; }
public void Dispose() => Provider.Dispose();
}
/// <summary>
/// LDAP accepts alice/password (groups ["Domain Users"]) and rejects alice/wrong; the role mapper
/// answers per <paramref name="mapping"/>, with <paramref name="oneRole"/> as the single role.
/// </summary>
private static Rig BuildRig(Mapping mapping, string oneRole, int maxFailures = 5)
{
var logs = new CapturingLoggerProvider();
var services = new ServiceCollection();
services.AddLogging(b => b.AddProvider(logs));
services.AddSingleton<IOptions<AuthDisableLoginOptions>>(
Options.Create(new AuthDisableLoginOptions { DisableLogin = false }));
services.AddSingleton<IOptions<SecurityOptions>>(Options.Create(new SecurityOptions
{
JwtSigningKey = Key,
JwtExpiryMinutes = 15,
MaxLoginFailuresPerWindow = maxFailures,
PerIpMaxFailures = 25,
}));
services.AddSingleton(TimeProvider.System);
services.AddSingleton<LoginThrottle>();
services.AddScoped<JwtTokenService>();
var ldap = Substitute.For<ILdapAuthService>();
ldap.AuthenticateAsync("alice", "password", Arg.Any<CancellationToken>())
.Returns(LdapAuthResult.Success("alice", "Alice", ["Domain Users"]));
ldap.AuthenticateAsync("alice", "wrong", Arg.Any<CancellationToken>())
.Returns(LdapAuthResult.Fail(LdapAuthFailure.BadCredentials));
services.AddSingleton(ldap);
var roleMapper = Substitute.For<RoleMapper>(Substitute.For<ISecurityRepository>());
var call = roleMapper.MapGroupsToRolesAsync(Arg.Any<IReadOnlyList<string>>(), Arg.Any<CancellationToken>());
switch (mapping)
{
case Mapping.Throws:
call.ThrowsAsync(new InvalidOperationException("configuration database unreachable"));
break;
case Mapping.ZeroRoles:
call.Returns(new RoleMappingResult([], [], IsSystemWideDeployment: false));
break;
default:
call.Returns(new RoleMappingResult([oneRole], [], IsSystemWideDeployment: true));
break;
}
services.AddSingleton(roleMapper);
// /management: no actor → a request that passes authentication and parsing answers 503.
services.AddSingleton(new ManagementActorHolder());
var repo = Substitute.For<IAuditLogRepository>();
repo.QueryAsync(Arg.Any<AuditLogQueryFilter>(), Arg.Any<AuditLogPaging>(), Arg.Any<CancellationToken>())
.Returns(Task.FromResult<IReadOnlyList<AuditEvent>>(Array.Empty<AuditEvent>()));
services.AddSingleton(repo);
var provider = services.BuildServiceProvider();
return new Rig
{
Provider = provider,
Throttle = provider.GetRequiredService<LoginThrottle>(),
Logs = logs,
RoleMapper = roleMapper,
};
}
private static string Basic(string password) =>
"Basic " + Convert.ToBase64String(Encoding.UTF8.GetBytes($"alice:{password}"));
private static string Bearer(params string[] roles)
{
var jwt = new JwtTokenService(
Options.Create(new SecurityOptions { JwtSigningKey = Key, JwtExpiryMinutes = 15 }),
NullLogger<JwtTokenService>.Instance);
return "Bearer " + jwt.GenerateToken("Alice", "alice", roles, null);
}
private static DefaultHttpContext Request(Rig rig, string path, string authorization)
{
var context = new DefaultHttpContext { RequestServices = rig.Provider };
context.Request.Path = path;
context.Request.Headers.Authorization = authorization;
context.Connection.RemoteIpAddress = IPAddress.Parse(Ip);
context.Response.Body = new MemoryStream();
return context;
}
private static async Task<(int Status, string Body)> Execute(HttpContext context, IResult result)
{
await result.ExecuteAsync(context);
context.Response.Body.Position = 0;
using var reader = new StreamReader(context.Response.Body);
return (context.Response.StatusCode, await reader.ReadToEndAsync());
}
private static async Task<(int Status, string Body)> PostManagement(Rig rig, string authorization, string command = "ListSites")
{
var context = Request(rig, "/management", authorization);
context.Request.Method = "POST";
var body = Encoding.UTF8.GetBytes("{\"command\":\"" + command + "\",\"payload\":{}}");
context.Request.Body = new MemoryStream(body);
context.Request.ContentLength = body.Length;
return await Execute(context, await ManagementEndpoints.HandleRequest(context));
}
private static async Task<(int Status, string Body)> GetAuditQuery(Rig rig, string authorization)
{
var context = Request(rig, "/api/audit/query", authorization);
context.Request.Method = "GET";
return await Execute(context, await AuditEndpoints.HandleQuery(context));
}
/// <summary>The 401 body a bad password gets on the same surface — every Basic refusal must match it.</summary>
private static async Task<(int Status, string Body)> BadPassword(Func<Rig, string, Task<(int, string)>> send, string oneRole)
{
using var rig = BuildRig(Mapping.OneRole, oneRole);
return await send(rig, Basic("wrong"));
}
private static CapturingLoggerProvider.Entry SingleRefusal(Rig rig) =>
Assert.Single(rig.Logs.Entries, e => e.Category == Category && e.Level == LogLevel.Warning);
private static void AssertBasicRefusalLogged(Rig rig, Mapping mapping, string endpoint)
{
var entry = SingleRefusal(rig);
Assert.Contains("alice", entry.Message);
Assert.Contains(endpoint, entry.Message);
if (mapping == Mapping.Throws)
{
Assert.Contains("role mapping failed", entry.Message);
Assert.IsType<InvalidOperationException>(entry.Exception);
}
else
{
Assert.Contains("maps to a ScadaBridge role", entry.Message);
}
}
private static void AssertNoRefusalLogged(Rig rig) =>
Assert.DoesNotContain(rig.Logs.Entries, e => e.Category == Category && e.Level >= LogLevel.Warning);
// ── POST /management (CLI) ──────────────────────────────────────────────────────────────
/// <summary>The command these tests send is one <c>GetRequiredRoles</c> opens to any authenticated user.</summary>
[Fact]
public void ListSites_IsOpenToAnyAuthenticatedUser()
{
Assert.Null(ManagementActor.GetRequiredRoles(new ListSitesCommand()));
}
/// <summary>A zero-role or mapper-fault Basic bind gets the bad-password 401 on a null-role command, and it is logged.</summary>
/// <param name="mapping">The refused mapping outcome.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
[Theory]
[InlineData(Mapping.ZeroRoles)]
[InlineData(Mapping.Throws)]
public async Task Management_Basic_RefusedLikeABadPassword(Mapping mapping)
{
using var rig = BuildRig(mapping, Roles.Viewer);
var (status, body) = await PostManagement(rig, Basic("password"));
var badPassword = await BadPassword((r, a) => PostManagement(r, a), Roles.Viewer);
Assert.Equal(401, status);
Assert.Equal(badPassword, (status, body));
Assert.Contains("AUTH_FAILED", body);
Assert.Contains(LdapAuthFailureMessages.InvalidCredentials, body);
AssertBasicRefusalLogged(rig, mapping, "/management");
}
/// <summary>A one-role Basic bind passes authentication and reaches dispatch (503: no actor in the rig).</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Management_Basic_OneRole_ReachesDispatch()
{
using var rig = BuildRig(Mapping.OneRole, Roles.Viewer);
var (status, _) = await PostManagement(rig, Basic("password"));
Assert.Equal(503, status);
AssertNoRefusalLogged(rig);
}
/// <summary>A bearer token with no role claim is refused with the invalid-token 401, and it is logged.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Management_Bearer_ZeroRoles_Refused()
{
using var rig = BuildRig(Mapping.OneRole, Roles.Viewer);
var (status, body) = await PostManagement(rig, Bearer());
var (_, invalidTokenBody) = await PostManagement(rig, "Bearer not-a-token");
Assert.Equal(401, status);
Assert.Equal(invalidTokenBody, body);
var entry = SingleRefusal(rig);
Assert.Contains("alice", entry.Message);
Assert.Contains("/management", entry.Message);
Assert.Contains("no ScadaBridge role", entry.Message);
}
/// <summary>
/// Bearer performs no role mapping: a one-role token is accepted even while the mapper throws, and
/// the mapper is never called. (The mapper-fault refusal is a Basic-path rule.)
/// </summary>
/// <param name="mapping">The mapper state, which a bearer request must not consult.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
[Theory]
[InlineData(Mapping.OneRole)]
[InlineData(Mapping.Throws)]
public async Task Management_Bearer_OneRole_ReachesDispatch_WithoutMapping(Mapping mapping)
{
using var rig = BuildRig(mapping, Roles.Viewer);
var (status, _) = await PostManagement(rig, Bearer(Roles.Viewer));
Assert.Equal(503, status);
AssertNoRefusalLogged(rig);
await rig.RoleMapper.DidNotReceiveWithAnyArgs().MapGroupsToRolesAsync(default!, default);
}
/// <summary>Every management command <c>GetRequiredRoles</c> opens to any authenticated user.</summary>
/// <returns>The theory data rows of command names.</returns>
public static IEnumerable<object[]> NullRoleCommands()
=> typeof(ManagementEnvelope).Assembly.GetTypes()
.Where(t => t.Namespace == typeof(ManagementEnvelope).Namespace
&& t.Name.EndsWith("Command", StringComparison.Ordinal) && !t.IsAbstract
&& ManagementActor.GetRequiredRoles(RuntimeHelpers.GetUninitializedObject(t)) is null)
.Select(t => new object[] { t.Name[..^"Command".Length] });
/// <summary>
/// No command open to any authenticated user is reachable by a zero-role principal, over Basic or
/// Bearer: authentication answers 401 before the command is parsed or dispatched.
/// </summary>
/// <param name="command">A null-role command name.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
[Theory, MemberData(nameof(NullRoleCommands))]
public async Task NullRoleCommand_IsNotReachableByAZeroRolePrincipal(string command)
{
using var rig = BuildRig(Mapping.ZeroRoles, Roles.Viewer);
var (basicStatus, _) = await PostManagement(rig, Basic("password"), command);
var (bearerStatus, _) = await PostManagement(rig, Bearer(), command);
Assert.Equal(401, basicStatus);
Assert.Equal(401, bearerStatus);
}
/// <summary>A zero-role Basic refusal counts against the login throttle like a bad password.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Management_Basic_ZeroRoles_CountsAsAThrottleFailure()
{
using var rig = BuildRig(Mapping.ZeroRoles, Roles.Viewer, maxFailures: 2);
await PostManagement(rig, Basic("password"));
await PostManagement(rig, Basic("password"));
Assert.True(rig.Throttle.IsLockedOut("alice", Ip));
var (status, body) = await PostManagement(rig, Basic("password"));
Assert.Equal(429, status);
Assert.Contains("AUTH_THROTTLED", body);
}
/// <summary>
/// A refusal does not clear earlier failures the way a successful sign-in does: a bad password
/// followed by a zero-role bind leaves both failures on the throttle.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Management_Basic_ZeroRoles_DoesNotResetTheThrottle()
{
using var rig = BuildRig(Mapping.ZeroRoles, Roles.Viewer, maxFailures: 2);
await PostManagement(rig, Basic("wrong"));
await PostManagement(rig, Basic("password"));
Assert.True(rig.Throttle.IsLockedOut("alice", Ip));
}
// ── GET /api/audit/query ────────────────────────────────────────────────────────────────
/// <summary>A zero-role or mapper-fault Basic bind gets the bad-password 401 on an audit query (not a 403), and it is logged.</summary>
/// <param name="mapping">The refused mapping outcome.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
[Theory]
[InlineData(Mapping.ZeroRoles)]
[InlineData(Mapping.Throws)]
public async Task Audit_Basic_RefusedLikeABadPassword(Mapping mapping)
{
using var rig = BuildRig(mapping, Roles.Administrator);
var (status, body) = await GetAuditQuery(rig, Basic("password"));
var badPassword = await BadPassword((r, a) => GetAuditQuery(r, a), Roles.Administrator);
Assert.Equal(401, status);
Assert.Equal(badPassword, (status, body));
AssertBasicRefusalLogged(rig, mapping, "/api/audit/query");
}
/// <summary>A one-role Basic bind with an audit role gets the query page.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Audit_Basic_OneRole_Answers()
{
using var rig = BuildRig(Mapping.OneRole, Roles.Administrator);
var (status, _) = await GetAuditQuery(rig, Basic("password"));
Assert.Equal(200, status);
AssertNoRefusalLogged(rig);
}
/// <summary>A bearer token with no role claim gets 401 on an audit query (not a 403), and it is logged.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Audit_Bearer_ZeroRoles_Refused()
{
using var rig = BuildRig(Mapping.OneRole, Roles.Administrator);
var (status, body) = await GetAuditQuery(rig, Bearer());
Assert.Equal(401, status);
Assert.Contains("AUTH_FAILED", body);
Assert.Contains("/api/audit/query", SingleRefusal(rig).Message);
}
/// <summary>A one-role bearer token gets the query page without any role mapping, even while the mapper throws.</summary>
/// <param name="mapping">The mapper state, which a bearer request must not consult.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
[Theory]
[InlineData(Mapping.OneRole)]
[InlineData(Mapping.Throws)]
public async Task Audit_Bearer_OneRole_Answers(Mapping mapping)
{
using var rig = BuildRig(mapping, Roles.Administrator);
var (status, _) = await GetAuditQuery(rig, Bearer(Roles.Administrator));
Assert.Equal(200, status);
AssertNoRefusalLogged(rig);
}
// ── /hubs/debug-stream ──────────────────────────────────────────────────────────────────
private static async Task<TestHubCallerContext> ConnectHub(Rig rig, string authorization)
{
var http = Request(rig, "/hubs/debug-stream", authorization);
var debugStreamService = new DebugStreamService(
new CommunicationService(Options.Create(new CommunicationOptions()), NullLogger<CommunicationService>.Instance),
new ServiceCollection().BuildServiceProvider(),
new SiteStreamGrpcClientFactory(NullLoggerFactory.Instance),
NullLogger<DebugStreamService>.Instance);
var hub = new DebugStreamHub(debugStreamService, Substitute.For<IHubContext<DebugStreamHub>>(),
NullLogger<DebugStreamHub>.Instance);
var ctx = new TestHubCallerContext(http);
hub.Context = ctx;
await hub.OnConnectedAsync();
return ctx;
}
/// <summary>
/// A zero-role or mapper-fault Basic connect is aborted by the shared authenticator — logged as a
/// sign-in refusal and counted as a throttle failure — not merely by the hub's Deployer check.
/// </summary>
/// <param name="mapping">The refused mapping outcome.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
[Theory]
[InlineData(Mapping.ZeroRoles)]
[InlineData(Mapping.Throws)]
public async Task Hub_Basic_Refused(Mapping mapping)
{
using var rig = BuildRig(mapping, Roles.Deployer, maxFailures: 1);
var ctx = await ConnectHub(rig, Basic("password"));
Assert.True(ctx.Aborted);
AssertBasicRefusalLogged(rig, mapping, "/hubs/debug-stream");
Assert.True(rig.Throttle.IsLockedOut("alice", Ip));
}
/// <summary>A one-role (Deployer) Basic connect is established.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Hub_Basic_OneRole_Connects()
{
using var rig = BuildRig(Mapping.OneRole, Roles.Deployer);
var ctx = await ConnectHub(rig, Basic("password"));
Assert.False(ctx.Aborted);
AssertNoRefusalLogged(rig);
}
/// <summary>A bearer token with no role claim is refused by the shared authenticator, and it is logged.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Hub_Bearer_ZeroRoles_Refused()
{
using var rig = BuildRig(Mapping.OneRole, Roles.Deployer);
var ctx = await ConnectHub(rig, Bearer());
Assert.True(ctx.Aborted);
var entry = SingleRefusal(rig);
Assert.Contains("/hubs/debug-stream", entry.Message);
Assert.Contains("no ScadaBridge role", entry.Message);
}
/// <summary>A one-role (Deployer) bearer connect is established without role mapping, even while the mapper throws.</summary>
/// <param name="mapping">The mapper state, which a bearer request must not consult.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
[Theory]
[InlineData(Mapping.OneRole)]
[InlineData(Mapping.Throws)]
public async Task Hub_Bearer_OneRole_Connects(Mapping mapping)
{
using var rig = BuildRig(mapping, Roles.Deployer);
var ctx = await ConnectHub(rig, Bearer(Roles.Deployer));
Assert.False(ctx.Aborted);
AssertNoRefusalLogged(rig);
}
// ── DisableLogin stays unchanged ────────────────────────────────────────────────────────
/// <summary>The dev/test bypass still admits every surface with all roles, without LDAP or role mapping.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task DisableLogin_StillBypasses_WithAllRoles()
{
var services = new ServiceCollection();
services.AddSingleton<IOptions<AuthDisableLoginOptions>>(
Options.Create(new AuthDisableLoginOptions { DisableLogin = true }));
using var provider = services.BuildServiceProvider();
var outcome = await ManagementAuthenticator.AuthenticateAsync(new DefaultHttpContext { RequestServices = provider });
Assert.Null(outcome.Failure);
Assert.Equal(Roles.All, outcome.User!.Roles);
}
private sealed class TestHubCallerContext : HubCallerContext
{
private readonly FeatureCollection _features = new();
public TestHubCallerContext(HttpContext http) =>
_features.Set<IHttpContextFeature>(new TestHttpContextFeature { HttpContext = http });
public bool Aborted { get; private set; }
public override string ConnectionId => "test-conn";
public override string? UserIdentifier => null;
public override System.Security.Claims.ClaimsPrincipal? User => null;
public override IDictionary<object, object?> Items { get; } = new Dictionary<object, object?>();
public override IFeatureCollection Features => _features;
public override CancellationToken ConnectionAborted => CancellationToken.None;
public override void Abort() => Aborted = true;
private sealed class TestHttpContextFeature : IHttpContextFeature
{
public HttpContext? HttpContext { get; set; }
}
}
}