From 0289d511376279be94829b1df9fda35f74606f84 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 28 Sep 2026 21:33:10 -0400 Subject: [PATCH 1/2] fix(site-secrets): report every site node's secret mirror failures on the health report (M1) A DeployedSecretsMirror failure on the STANDBY node (e.g. a master-key mismatch that stops a deployed secret decrypting there) was only logged on that node: site health reports are sent by the active node only, so central never saw it. - New 12th LocalDb-replicated table secret_mirror_status (node_name PK, failures_json of {SecretName, Message}, checked_at; no FKs). Each node's mirror writes only its own row, and only when its failure set changes plus once per start-up (every write replicates via CDC). Best-effort: a failed write is a Warning and is retried next pass. Node key = INodeIdentityProvider NodeName, falling back to the machine name. - SecretMirrorStatusReporter (SiteRuntime, every node, 30 s) reads every row into ISiteHealthCollector.SetSecretMirrorStatuses -> additive nullable SiteHealthReport.SecretMirrorStatuses (null = no data). - Wire: SiteHealthReportDto.secret_mirror_statuses = 33 (presence wrapper so null != empty); vendored CentralControl.cs regenerated; mapper both ways. - Central UI Health card: 'Secret mirror failing' badge + compact per-node node / secret / message list. - Tests: mirror status write-on-change/first-pass/failure isolation, storage round-trip, reporter/collector, DTO mapper round-trip incl. null/empty, bUnit, replicated-table set (twelve) + FK guard wording. --- .../Components/Pages/Monitoring/Health.razor | 37 + .../Messages/Health/SecretMirrorNodeStatus.cs | 26 + .../Messages/Health/SiteHealthReport.cs | 14 + .../CentralControlGrpc/CentralControl.cs | 919 ++++++++++++++++-- .../Grpc/CentralControlDtoMapper.cs | 40 + .../Protos/central_control.proto | 23 + .../ISiteHealthCollector.cs | 13 + .../SiteHealthCollector.cs | 14 +- .../SiteLocalDbSetup.cs | 15 +- .../Persistence/SiteStorageSchema.cs | 20 +- .../Persistence/SiteStorageService.cs | 82 ++ .../Secrets/DeployedSecretsMirror.cs | 72 +- .../Secrets/SecretMirrorStatusReporter.cs | 95 ++ .../ServiceCollectionExtensions.cs | 7 + .../Pages/HealthPageTests.cs | 64 ++ .../Grpc/CentralControlDtoMapperTests.cs | 28 + .../SecretMirrorStatusHealthFieldTests.cs | 45 + .../SiteLocalDbCdcRegistrationTests.cs | 6 +- .../SiteLocalDbReplicatedTableFkGuardTests.cs | 5 +- .../SiteLocalDbWiringTests.cs | 12 +- .../DeployedSecretsStorageTests.cs | 49 + .../Persistence/SiteStorageSchemaTests.cs | 4 +- .../Secrets/DeployedSecretsMirrorTests.cs | 133 ++- .../SecretMirrorStatusReporterTests.cs | 95 ++ 24 files changed, 1718 insertions(+), 100 deletions(-) create mode 100644 src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SecretMirrorNodeStatus.cs create mode 100644 src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Secrets/SecretMirrorStatusReporter.cs create mode 100644 tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/SecretMirrorStatusHealthFieldTests.cs create mode 100644 tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Secrets/SecretMirrorStatusReporterTests.cs diff --git a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/Health.razor b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/Health.razor index 8ecaed72..487268d7 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/Health.razor +++ b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/Health.razor @@ -275,6 +275,14 @@ No self-heal } + @if (HasSecretMirrorFailures(state)) + { + + Secret mirror failing + + } @if (!state.IsOnline && state.LastStatusChangeAt is { } changedAt) { offline since @changedAt.ToString("u") @@ -302,6 +310,26 @@ @if (state.LatestReport != null) { var report = state.LatestReport; + @if (HasSecretMirrorFailures(state)) + { + @* Review finding M1: every site node's deployed-secrets mirror failures, + the standby's included (carried on the active node's report). *@ + + }
@* Each logical group below sits in its own card, with the group's head promoted to a real card-header, instead of h6 rules stacked @@ -590,6 +618,15 @@ state.IsOnline && state.LatestReport?.StartupReconcilerActive == false; + // Review finding M1: warn when any site node (the standby included) reports deployed + // secrets it cannot apply/decrypt. Null ("no data": central self-report, pre-M1 site) and + // nodes with an empty failure list are healthy. Offline sites are skipped like the other + // warning badges — the Offline badge is the signal that matters there. + private static bool HasSecretMirrorFailures(SiteHealthState state) => + state.IsOnline + && state.LatestReport?.SecretMirrorStatuses is { } statuses + && statuses.Any(n => n.Failures.Count > 0); + // Online central nodes, from the same ClusterNodes list the Nodes column renders. // Fewer than two means no standby, which disables the manual-failover control. This is // a display-side guard only — AkkaManualFailoverService re-checks against live cluster diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SecretMirrorNodeStatus.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SecretMirrorNodeStatus.cs new file mode 100644 index 00000000..094c750b --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SecretMirrorNodeStatus.cs @@ -0,0 +1,26 @@ +namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.Health; + +/// +/// One site node's latest deployed-secrets mirror outcome, as carried on +/// (review finding M1). +/// +/// +/// Every site node runs its own DeployedSecretsMirror and records its current +/// failure set in the LocalDb-replicated secret_mirror_status table, keyed by node +/// name; the ACTIVE node reads every row and ships them on its health report, so a +/// STANDBY node's failures (most importantly a master-key mismatch that stops a deployed +/// secret decrypting there) reach central before a failover exposes them. Carries names +/// and human-readable messages only — never a secret value or ciphertext. +/// +/// The reporting node's semantic name (node-a/node-b), or its host name when unconfigured. +/// When that node last recorded a CHANGE in its failure set (or its first pass after start-up) — not the time of its latest pass. +/// That node's current mirror failures; empty when the node's mirror is healthy. +public sealed record SecretMirrorNodeStatus( + string NodeName, + DateTimeOffset CheckedAtUtc, + IReadOnlyList Failures); + +/// A single deployed secret that failed to apply and/or decrypt on one site node. +/// The secret's name. +/// A human-readable failure message — never includes the secret's value. +public sealed record SecretMirrorFailure(string SecretName, string Message); diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SiteHealthReport.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SiteHealthReport.cs index 346dd034..7525ebd0 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SiteHealthReport.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SiteHealthReport.cs @@ -173,6 +173,20 @@ public record SiteHealthReport( /// /// public bool? StartupReconcilerActive { get; init; } + + /// + /// Every site node's latest deployed-secrets mirror outcome (review finding M1) — one + /// entry per node that has recorded a row in the replicated secret_mirror_status + /// table, including the STANDBY node, whose own health report is never sent. + /// + /// + /// Null = no data: a central self-report, a pre-M1 site build, a site whose + /// reporter has not run yet, or a failed read. An EMPTY list means the feature is wired + /// but no node has recorded a status yet. A node entry with an empty + /// list is a healthy mirror. Names and + /// messages only — never secret values or ciphertext. + /// + public IReadOnlyList? SecretMirrorStatuses { get; init; } } /// diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/CentralControlGrpc/CentralControl.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/CentralControlGrpc/CentralControl.cs index 9e3f8b48..48fd90fe 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/CentralControlGrpc/CentralControl.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/CentralControlGrpc/CentralControl.cs @@ -72,80 +72,89 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { "ZXkYASABKAkSQQoFdmFsdWUYAiABKAsyMi5zY2FkYWJyaWRnZS5jZW50cmFs", "Y29udHJvbC52MS5UYWdRdWFsaXR5Q291bnRzRHRvOgI4ASJQChFOb2RlU3Rh", "dHVzTGlzdER0bxI7CgVub2RlcxgBIAMoCzIsLnNjYWRhYnJpZGdlLmNlbnRy", - "YWxjb250cm9sLnYxLk5vZGVTdGF0dXNEdG8irw8KE1NpdGVIZWFsdGhSZXBv", - "cnREdG8SDwoHc2l0ZV9pZBgBIAEoCRIXCg9zZXF1ZW5jZV9udW1iZXIYAiAB", - "KAMSNAoQcmVwb3J0X3RpbWVzdGFtcBgDIAEoCzIaLmdvb2dsZS5wcm90b2J1", - "Zi5UaW1lc3RhbXAScAoYZGF0YV9jb25uZWN0aW9uX3N0YXR1c2VzGAQgAygL", - "Mk4uc2NhZGFicmlkZ2UuY2VudHJhbGNvbnRyb2wudjEuU2l0ZUhlYWx0aFJl", - "cG9ydER0by5EYXRhQ29ubmVjdGlvblN0YXR1c2VzRW50cnkSagoVdGFnX3Jl", - "c29sdXRpb25fY291bnRzGAUgAygLMksuc2NhZGFicmlkZ2UuY2VudHJhbGNv", - "bnRyb2wudjEuU2l0ZUhlYWx0aFJlcG9ydER0by5UYWdSZXNvbHV0aW9uQ291", - "bnRzRW50cnkSGgoSc2NyaXB0X2Vycm9yX2NvdW50GAYgASgFEiQKHGFsYXJt", - "X2V2YWx1YXRpb25fZXJyb3JfY291bnQYByABKAUSfAofc3RvcmVfYW5kX2Zv", - "cndhcmRfYnVmZmVyX2RlcHRocxgIIAMoCzJTLnNjYWRhYnJpZGdlLmNlbnRy", - "YWxjb250cm9sLnYxLlNpdGVIZWFsdGhSZXBvcnREdG8uU3RvcmVBbmRGb3J3", - "YXJkQnVmZmVyRGVwdGhzRW50cnkSGQoRZGVhZF9sZXR0ZXJfY291bnQYCSAB", - "KAUSHwoXZGVwbG95ZWRfaW5zdGFuY2VfY291bnQYCiABKAUSHgoWZW5hYmxl", - "ZF9pbnN0YW5jZV9jb3VudBgLIAEoBRIfChdkaXNhYmxlZF9pbnN0YW5jZV9j", - "b3VudBgMIAEoBRIRCglub2RlX3JvbGUYDSABKAkSFQoNbm9kZV9ob3N0bmFt", - "ZRgOIAEoCRJaChlkYXRhX2Nvbm5lY3Rpb25fZW5kcG9pbnRzGA8gASgLMjcu", - "c2NhZGFicmlkZ2UuY2VudHJhbGNvbnRyb2wudjEuQ29ubmVjdGlvbkVuZHBv", - "aW50TWFwRHRvElQKG2RhdGFfY29ubmVjdGlvbl90YWdfcXVhbGl0eRgQIAEo", - "CzIvLnNjYWRhYnJpZGdlLmNlbnRyYWxjb250cm9sLnYxLlRhZ1F1YWxpdHlN", - "YXBEdG8SHAoUcGFya2VkX21lc3NhZ2VfY291bnQYESABKAUSRwoNY2x1c3Rl", - "cl9ub2RlcxgSIAEoCzIwLnNjYWRhYnJpZGdlLmNlbnRyYWxjb250cm9sLnYx", - "Lk5vZGVTdGF0dXNMaXN0RHRvEiEKGXNpdGVfYXVkaXRfd3JpdGVfZmFpbHVy", - "ZXMYEyABKAUSHwoXYXVkaXRfcmVkYWN0aW9uX2ZhaWx1cmUYFCABKAUSVgoS", - "c2l0ZV9hdWRpdF9iYWNrbG9nGBUgASgLMjouc2NhZGFicmlkZ2UuY2VudHJh", - "bGNvbnRyb2wudjEuU2l0ZUF1ZGl0QmFja2xvZ1NuYXBzaG90RHRvEiUKHXNp", - "dGVfZXZlbnRfbG9nX3dyaXRlX2ZhaWx1cmVzGBYgASgDEkcKIW9sZGVzdF9w", - "YXJrZWRfbWVzc2FnZV9hZ2Vfc2Vjb25kcxgXIAEoCzIcLmdvb2dsZS5wcm90", - "b2J1Zi5Eb3VibGVWYWx1ZRIaChJzY3JpcHRfcXVldWVfZGVwdGgYGCABKAUS", - "GwoTc2NyaXB0X2J1c3lfdGhyZWFkcxgZIAEoBRJECh5zY3JpcHRfb2xkZXN0", - "X2J1c3lfYWdlX3NlY29uZHMYGiABKAsyHC5nb29nbGUucHJvdG9idWYuRG91", - "YmxlVmFsdWUSQgoebG9jYWxfZGJfcmVwbGljYXRpb25fY29ubmVjdGVkGBsg", - "ASgLMhouZ29vZ2xlLnByb3RvYnVmLkJvb2xWYWx1ZRI7ChZsb2NhbF9kYl9v", - "cGxvZ19iYWNrbG9nGBwgASgLMhsuZ29vZ2xlLnByb3RvYnVmLkludDY0VmFs", - "dWUSPQoZc3RhcnR1cF9yZWNvbmNpbGVyX2FjdGl2ZRgdIAEoCzIaLmdvb2ds", - "ZS5wcm90b2J1Zi5Cb29sVmFsdWUSHwoXZGV0YWNoZWRfc2NyaXB0X3RocmVh", - "ZHMYHiABKAUSHQoVc2NyaXB0X3J1bl9zaGVkX2NvdW50GB8gASgFEiQKHHNp", - "dGVfc3RyZWFtX2FsYXJtX2Ryb3BfY291bnQYICABKAMacgobRGF0YUNvbm5l", - "Y3Rpb25TdGF0dXNlc0VudHJ5EgsKA2tleRgBIAEoCRJCCgV2YWx1ZRgCIAEo", - "DjIzLnNjYWRhYnJpZGdlLmNlbnRyYWxjb250cm9sLnYxLkNvbm5lY3Rpb25I", - "ZWFsdGhFbnVtOgI4ARpxChhUYWdSZXNvbHV0aW9uQ291bnRzRW50cnkSCwoD", - "a2V5GAEgASgJEkQKBXZhbHVlGAIgASgLMjUuc2NhZGFicmlkZ2UuY2VudHJh", - "bGNvbnRyb2wudjEuVGFnUmVzb2x1dGlvblN0YXR1c0R0bzoCOAEaQgogU3Rv", - "cmVBbmRGb3J3YXJkQnVmZmVyRGVwdGhzRW50cnkSCwoDa2V5GAEgASgJEg0K", - "BXZhbHVlGAIgASgFOgI4ASJjChZTaXRlSGVhbHRoUmVwb3J0QWNrRHRvEg8K", - "B3NpdGVfaWQYASABKAkSFwoPc2VxdWVuY2VfbnVtYmVyGAIgASgDEhAKCGFj", - "Y2VwdGVkGAMgASgIEg0KBWVycm9yGAQgASgJIosBCgxIZWFydGJlYXREdG8S", - "DwoHc2l0ZV9pZBgBIAEoCRIVCg1ub2RlX2hvc3RuYW1lGAIgASgJEhEKCWlz", - "X2FjdGl2ZRgDIAEoCBItCgl0aW1lc3RhbXAYBCABKAsyGi5nb29nbGUucHJv", - "dG9idWYuVGltZXN0YW1wEhEKCXN5bnRoZXRpYxgFIAEoCCq9AQoUQ29ubmVj", - "dGlvbkhlYWx0aEVudW0SIQodQ09OTkVDVElPTl9IRUFMVEhfVU5TUEVDSUZJ", - "RUQQABIfChtDT05ORUNUSU9OX0hFQUxUSF9DT05ORUNURUQQARIiCh5DT05O", - "RUNUSU9OX0hFQUxUSF9ESVNDT05ORUNURUQQAhIgChxDT05ORUNUSU9OX0hF", - "QUxUSF9DT05ORUNUSU5HEAMSGwoXQ09OTkVDVElPTl9IRUFMVEhfRVJST1IQ", - "BDKhBgoVQ2VudHJhbENvbnRyb2xTZXJ2aWNlEoMBChJTdWJtaXROb3RpZmlj", - "YXRpb24SNC5zY2FkYWJyaWRnZS5jZW50cmFsY29udHJvbC52MS5Ob3RpZmlj", - "YXRpb25TdWJtaXREdG8aNy5zY2FkYWJyaWRnZS5jZW50cmFsY29udHJvbC52", - "MS5Ob3RpZmljYXRpb25TdWJtaXRBY2tEdG8SkgEKF1F1ZXJ5Tm90aWZpY2F0", - "aW9uU3RhdHVzEjkuc2NhZGFicmlkZ2UuY2VudHJhbGNvbnRyb2wudjEuTm90", - "aWZpY2F0aW9uU3RhdHVzUXVlcnlEdG8aPC5zY2FkYWJyaWRnZS5jZW50cmFs", - "Y29udHJvbC52MS5Ob3RpZmljYXRpb25TdGF0dXNSZXNwb25zZUR0bxJHChFJ", - "bmdlc3RBdWRpdEV2ZW50cxIbLnNpdGVzdHJlYW0uQXVkaXRFdmVudEJhdGNo", - "GhUuc2l0ZXN0cmVhbS5Jbmdlc3RBY2sSUAoVSW5nZXN0Q2FjaGVkVGVsZW1l", - "dHJ5EiAuc2l0ZXN0cmVhbS5DYWNoZWRUZWxlbWV0cnlCYXRjaBoVLnNpdGVz", - "dHJlYW0uSW5nZXN0QWNrEoABCg1SZWNvbmNpbGVTaXRlEjYuc2NhZGFicmlk", - "Z2UuY2VudHJhbGNvbnRyb2wudjEuUmVjb25jaWxlU2l0ZVJlcXVlc3REdG8a", - "Ny5zY2FkYWJyaWRnZS5jZW50cmFsY29udHJvbC52MS5SZWNvbmNpbGVTaXRl", - "UmVzcG9uc2VEdG8SfQoQUmVwb3J0U2l0ZUhlYWx0aBIyLnNjYWRhYnJpZGdl", - "LmNlbnRyYWxjb250cm9sLnYxLlNpdGVIZWFsdGhSZXBvcnREdG8aNS5zY2Fk", - "YWJyaWRnZS5jZW50cmFsY29udHJvbC52MS5TaXRlSGVhbHRoUmVwb3J0QWNr", - "RHRvElAKCUhlYXJ0YmVhdBIrLnNjYWRhYnJpZGdlLmNlbnRyYWxjb250cm9s", - "LnYxLkhlYXJ0YmVhdER0bxoWLmdvb2dsZS5wcm90b2J1Zi5FbXB0eUIrqgIo", - "WkIuTU9NLldXLlNjYWRhQnJpZGdlLkNvbW11bmljYXRpb24uR3JwY2IGcHJv", - "dG8z")); + "YWxjb250cm9sLnYxLk5vZGVTdGF0dXNEdG8iPgoWU2VjcmV0TWlycm9yRmFp", + "bHVyZUR0bxITCgtzZWNyZXRfbmFtZRgBIAEoCRIPCgdtZXNzYWdlGAIgASgJ", + "IqsBChlTZWNyZXRNaXJyb3JOb2RlU3RhdHVzRHRvEhEKCW5vZGVfbmFtZRgB", + "IAEoCRIyCg5jaGVja2VkX2F0X3V0YxgCIAEoCzIaLmdvb2dsZS5wcm90b2J1", + "Zi5UaW1lc3RhbXASRwoIZmFpbHVyZXMYAyADKAsyNS5zY2FkYWJyaWRnZS5j", + "ZW50cmFsY29udHJvbC52MS5TZWNyZXRNaXJyb3JGYWlsdXJlRHRvImQKGVNl", + "Y3JldE1pcnJvclN0YXR1c0xpc3REdG8SRwoFbm9kZXMYASADKAsyOC5zY2Fk", + "YWJyaWRnZS5jZW50cmFsY29udHJvbC52MS5TZWNyZXRNaXJyb3JOb2RlU3Rh", + "dHVzRHRvIokQChNTaXRlSGVhbHRoUmVwb3J0RHRvEg8KB3NpdGVfaWQYASAB", + "KAkSFwoPc2VxdWVuY2VfbnVtYmVyGAIgASgDEjQKEHJlcG9ydF90aW1lc3Rh", + "bXAYAyABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEnAKGGRhdGFf", + "Y29ubmVjdGlvbl9zdGF0dXNlcxgEIAMoCzJOLnNjYWRhYnJpZGdlLmNlbnRy", + "YWxjb250cm9sLnYxLlNpdGVIZWFsdGhSZXBvcnREdG8uRGF0YUNvbm5lY3Rp", + "b25TdGF0dXNlc0VudHJ5EmoKFXRhZ19yZXNvbHV0aW9uX2NvdW50cxgFIAMo", + "CzJLLnNjYWRhYnJpZGdlLmNlbnRyYWxjb250cm9sLnYxLlNpdGVIZWFsdGhS", + "ZXBvcnREdG8uVGFnUmVzb2x1dGlvbkNvdW50c0VudHJ5EhoKEnNjcmlwdF9l", + "cnJvcl9jb3VudBgGIAEoBRIkChxhbGFybV9ldmFsdWF0aW9uX2Vycm9yX2Nv", + "dW50GAcgASgFEnwKH3N0b3JlX2FuZF9mb3J3YXJkX2J1ZmZlcl9kZXB0aHMY", + "CCADKAsyUy5zY2FkYWJyaWRnZS5jZW50cmFsY29udHJvbC52MS5TaXRlSGVh", + "bHRoUmVwb3J0RHRvLlN0b3JlQW5kRm9yd2FyZEJ1ZmZlckRlcHRoc0VudHJ5", + "EhkKEWRlYWRfbGV0dGVyX2NvdW50GAkgASgFEh8KF2RlcGxveWVkX2luc3Rh", + "bmNlX2NvdW50GAogASgFEh4KFmVuYWJsZWRfaW5zdGFuY2VfY291bnQYCyAB", + "KAUSHwoXZGlzYWJsZWRfaW5zdGFuY2VfY291bnQYDCABKAUSEQoJbm9kZV9y", + "b2xlGA0gASgJEhUKDW5vZGVfaG9zdG5hbWUYDiABKAkSWgoZZGF0YV9jb25u", + "ZWN0aW9uX2VuZHBvaW50cxgPIAEoCzI3LnNjYWRhYnJpZGdlLmNlbnRyYWxj", + "b250cm9sLnYxLkNvbm5lY3Rpb25FbmRwb2ludE1hcER0bxJUChtkYXRhX2Nv", + "bm5lY3Rpb25fdGFnX3F1YWxpdHkYECABKAsyLy5zY2FkYWJyaWRnZS5jZW50", + "cmFsY29udHJvbC52MS5UYWdRdWFsaXR5TWFwRHRvEhwKFHBhcmtlZF9tZXNz", + "YWdlX2NvdW50GBEgASgFEkcKDWNsdXN0ZXJfbm9kZXMYEiABKAsyMC5zY2Fk", + "YWJyaWRnZS5jZW50cmFsY29udHJvbC52MS5Ob2RlU3RhdHVzTGlzdER0bxIh", + "ChlzaXRlX2F1ZGl0X3dyaXRlX2ZhaWx1cmVzGBMgASgFEh8KF2F1ZGl0X3Jl", + "ZGFjdGlvbl9mYWlsdXJlGBQgASgFElYKEnNpdGVfYXVkaXRfYmFja2xvZxgV", + "IAEoCzI6LnNjYWRhYnJpZGdlLmNlbnRyYWxjb250cm9sLnYxLlNpdGVBdWRp", + "dEJhY2tsb2dTbmFwc2hvdER0bxIlCh1zaXRlX2V2ZW50X2xvZ193cml0ZV9m", + "YWlsdXJlcxgWIAEoAxJHCiFvbGRlc3RfcGFya2VkX21lc3NhZ2VfYWdlX3Nl", + "Y29uZHMYFyABKAsyHC5nb29nbGUucHJvdG9idWYuRG91YmxlVmFsdWUSGgoS", + "c2NyaXB0X3F1ZXVlX2RlcHRoGBggASgFEhsKE3NjcmlwdF9idXN5X3RocmVh", + "ZHMYGSABKAUSRAoec2NyaXB0X29sZGVzdF9idXN5X2FnZV9zZWNvbmRzGBog", + "ASgLMhwuZ29vZ2xlLnByb3RvYnVmLkRvdWJsZVZhbHVlEkIKHmxvY2FsX2Ri", + "X3JlcGxpY2F0aW9uX2Nvbm5lY3RlZBgbIAEoCzIaLmdvb2dsZS5wcm90b2J1", + "Zi5Cb29sVmFsdWUSOwoWbG9jYWxfZGJfb3Bsb2dfYmFja2xvZxgcIAEoCzIb", + "Lmdvb2dsZS5wcm90b2J1Zi5JbnQ2NFZhbHVlEj0KGXN0YXJ0dXBfcmVjb25j", + "aWxlcl9hY3RpdmUYHSABKAsyGi5nb29nbGUucHJvdG9idWYuQm9vbFZhbHVl", + "Eh8KF2RldGFjaGVkX3NjcmlwdF90aHJlYWRzGB4gASgFEh0KFXNjcmlwdF9y", + "dW5fc2hlZF9jb3VudBgfIAEoBRIkChxzaXRlX3N0cmVhbV9hbGFybV9kcm9w", + "X2NvdW50GCAgASgDElgKFnNlY3JldF9taXJyb3Jfc3RhdHVzZXMYISABKAsy", + "OC5zY2FkYWJyaWRnZS5jZW50cmFsY29udHJvbC52MS5TZWNyZXRNaXJyb3JT", + "dGF0dXNMaXN0RHRvGnIKG0RhdGFDb25uZWN0aW9uU3RhdHVzZXNFbnRyeRIL", + "CgNrZXkYASABKAkSQgoFdmFsdWUYAiABKA4yMy5zY2FkYWJyaWRnZS5jZW50", + "cmFsY29udHJvbC52MS5Db25uZWN0aW9uSGVhbHRoRW51bToCOAEacQoYVGFn", + "UmVzb2x1dGlvbkNvdW50c0VudHJ5EgsKA2tleRgBIAEoCRJECgV2YWx1ZRgC", + "IAEoCzI1LnNjYWRhYnJpZGdlLmNlbnRyYWxjb250cm9sLnYxLlRhZ1Jlc29s", + "dXRpb25TdGF0dXNEdG86AjgBGkIKIFN0b3JlQW5kRm9yd2FyZEJ1ZmZlckRl", + "cHRoc0VudHJ5EgsKA2tleRgBIAEoCRINCgV2YWx1ZRgCIAEoBToCOAEiYwoW", + "U2l0ZUhlYWx0aFJlcG9ydEFja0R0bxIPCgdzaXRlX2lkGAEgASgJEhcKD3Nl", + "cXVlbmNlX251bWJlchgCIAEoAxIQCghhY2NlcHRlZBgDIAEoCBINCgVlcnJv", + "chgEIAEoCSKLAQoMSGVhcnRiZWF0RHRvEg8KB3NpdGVfaWQYASABKAkSFQoN", + "bm9kZV9ob3N0bmFtZRgCIAEoCRIRCglpc19hY3RpdmUYAyABKAgSLQoJdGlt", + "ZXN0YW1wGAQgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcBIRCglz", + "eW50aGV0aWMYBSABKAgqvQEKFENvbm5lY3Rpb25IZWFsdGhFbnVtEiEKHUNP", + "Tk5FQ1RJT05fSEVBTFRIX1VOU1BFQ0lGSUVEEAASHwobQ09OTkVDVElPTl9I", + "RUFMVEhfQ09OTkVDVEVEEAESIgoeQ09OTkVDVElPTl9IRUFMVEhfRElTQ09O", + "TkVDVEVEEAISIAocQ09OTkVDVElPTl9IRUFMVEhfQ09OTkVDVElORxADEhsK", + "F0NPTk5FQ1RJT05fSEVBTFRIX0VSUk9SEAQyoQYKFUNlbnRyYWxDb250cm9s", + "U2VydmljZRKDAQoSU3VibWl0Tm90aWZpY2F0aW9uEjQuc2NhZGFicmlkZ2Uu", + "Y2VudHJhbGNvbnRyb2wudjEuTm90aWZpY2F0aW9uU3VibWl0RHRvGjcuc2Nh", + "ZGFicmlkZ2UuY2VudHJhbGNvbnRyb2wudjEuTm90aWZpY2F0aW9uU3VibWl0", + "QWNrRHRvEpIBChdRdWVyeU5vdGlmaWNhdGlvblN0YXR1cxI5LnNjYWRhYnJp", + "ZGdlLmNlbnRyYWxjb250cm9sLnYxLk5vdGlmaWNhdGlvblN0YXR1c1F1ZXJ5", + "RHRvGjwuc2NhZGFicmlkZ2UuY2VudHJhbGNvbnRyb2wudjEuTm90aWZpY2F0", + "aW9uU3RhdHVzUmVzcG9uc2VEdG8SRwoRSW5nZXN0QXVkaXRFdmVudHMSGy5z", + "aXRlc3RyZWFtLkF1ZGl0RXZlbnRCYXRjaBoVLnNpdGVzdHJlYW0uSW5nZXN0", + "QWNrElAKFUluZ2VzdENhY2hlZFRlbGVtZXRyeRIgLnNpdGVzdHJlYW0uQ2Fj", + "aGVkVGVsZW1ldHJ5QmF0Y2gaFS5zaXRlc3RyZWFtLkluZ2VzdEFjaxKAAQoN", + "UmVjb25jaWxlU2l0ZRI2LnNjYWRhYnJpZGdlLmNlbnRyYWxjb250cm9sLnYx", + "LlJlY29uY2lsZVNpdGVSZXF1ZXN0RHRvGjcuc2NhZGFicmlkZ2UuY2VudHJh", + "bGNvbnRyb2wudjEuUmVjb25jaWxlU2l0ZVJlc3BvbnNlRHRvEn0KEFJlcG9y", + "dFNpdGVIZWFsdGgSMi5zY2FkYWJyaWRnZS5jZW50cmFsY29udHJvbC52MS5T", + "aXRlSGVhbHRoUmVwb3J0RHRvGjUuc2NhZGFicmlkZ2UuY2VudHJhbGNvbnRy", + "b2wudjEuU2l0ZUhlYWx0aFJlcG9ydEFja0R0bxJQCglIZWFydGJlYXQSKy5z", + "Y2FkYWJyaWRnZS5jZW50cmFsY29udHJvbC52MS5IZWFydGJlYXREdG8aFi5n", + "b29nbGUucHJvdG9idWYuRW1wdHlCK6oCKFpCLk1PTS5XVy5TY2FkYUJyaWRn", + "ZS5Db21tdW5pY2F0aW9uLkdycGNiBnByb3RvMw==")); descriptor = pbr::FileDescriptor.FromGeneratedCode(descriptorData, new pbr::FileDescriptor[] { global::Google.Protobuf.WellKnownTypes.EmptyReflection.Descriptor, global::Google.Protobuf.WellKnownTypes.TimestampReflection.Descriptor, global::Google.Protobuf.WellKnownTypes.WrappersReflection.Descriptor, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor, }, new pbr::GeneratedClrTypeInfo(new[] {typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ConnectionHealthEnum), }, null, new pbr::GeneratedClrTypeInfo[] { @@ -163,7 +172,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ConnectionEndpointMapDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.ConnectionEndpointMapDto.Parser, new[]{ "Entries" }, null, null, null, new pbr::GeneratedClrTypeInfo[] { null, }), new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.TagQualityMapDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.TagQualityMapDto.Parser, new[]{ "Entries" }, null, null, null, new pbr::GeneratedClrTypeInfo[] { null, }), new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NodeStatusListDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.NodeStatusListDto.Parser, new[]{ "Nodes" }, null, null, null, null), - new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto.Parser, new[]{ "SiteId", "SequenceNumber", "ReportTimestamp", "DataConnectionStatuses", "TagResolutionCounts", "ScriptErrorCount", "AlarmEvaluationErrorCount", "StoreAndForwardBufferDepths", "DeadLetterCount", "DeployedInstanceCount", "EnabledInstanceCount", "DisabledInstanceCount", "NodeRole", "NodeHostname", "DataConnectionEndpoints", "DataConnectionTagQuality", "ParkedMessageCount", "ClusterNodes", "SiteAuditWriteFailures", "AuditRedactionFailure", "SiteAuditBacklog", "SiteEventLogWriteFailures", "OldestParkedMessageAgeSeconds", "ScriptQueueDepth", "ScriptBusyThreads", "ScriptOldestBusyAgeSeconds", "LocalDbReplicationConnected", "LocalDbOplogBacklog", "StartupReconcilerActive", "DetachedScriptThreads", "ScriptRunShedCount", "SiteStreamAlarmDropCount" }, null, null, null, new pbr::GeneratedClrTypeInfo[] { null, null, null, }), + new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SecretMirrorFailureDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SecretMirrorFailureDto.Parser, new[]{ "SecretName", "Message" }, null, null, null, null), + new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SecretMirrorNodeStatusDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SecretMirrorNodeStatusDto.Parser, new[]{ "NodeName", "CheckedAtUtc", "Failures" }, null, null, null, null), + new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SecretMirrorStatusListDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SecretMirrorStatusListDto.Parser, new[]{ "Nodes" }, null, null, null, null), + new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportDto.Parser, new[]{ "SiteId", "SequenceNumber", "ReportTimestamp", "DataConnectionStatuses", "TagResolutionCounts", "ScriptErrorCount", "AlarmEvaluationErrorCount", "StoreAndForwardBufferDepths", "DeadLetterCount", "DeployedInstanceCount", "EnabledInstanceCount", "DisabledInstanceCount", "NodeRole", "NodeHostname", "DataConnectionEndpoints", "DataConnectionTagQuality", "ParkedMessageCount", "ClusterNodes", "SiteAuditWriteFailures", "AuditRedactionFailure", "SiteAuditBacklog", "SiteEventLogWriteFailures", "OldestParkedMessageAgeSeconds", "ScriptQueueDepth", "ScriptBusyThreads", "ScriptOldestBusyAgeSeconds", "LocalDbReplicationConnected", "LocalDbOplogBacklog", "StartupReconcilerActive", "DetachedScriptThreads", "ScriptRunShedCount", "SiteStreamAlarmDropCount", "SecretMirrorStatuses" }, null, null, null, new pbr::GeneratedClrTypeInfo[] { null, null, null, }), new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteHealthReportAckDto.Parser, new[]{ "SiteId", "SequenceNumber", "Accepted", "Error" }, null, null, null, null), new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.HeartbeatDto.Parser, new[]{ "SiteId", "NodeHostname", "IsActive", "Timestamp", "Synthetic" }, null, null, null, null) })); @@ -4203,6 +4215,706 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { } + /// + /// Review finding M1: one site node's deployed-secrets mirror failures. Names and + /// human-readable messages only — never a secret value or ciphertext. + /// + [global::System.Diagnostics.DebuggerDisplayAttribute("{ToString(),nq}")] + public sealed partial class SecretMirrorFailureDto : pb::IMessage + #if !GOOGLE_PROTOBUF_REFSTRUCT_COMPATIBILITY_MODE + , pb::IBufferMessage + #endif + { + private static readonly pb::MessageParser _parser = new pb::MessageParser(() => new SecretMirrorFailureDto()); + private pb::UnknownFieldSet _unknownFields; + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public static pb::MessageParser Parser { get { return _parser; } } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public static pbr::MessageDescriptor Descriptor { + get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CentralControlReflection.Descriptor.MessageTypes[14]; } + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + pbr::MessageDescriptor pb::IMessage.Descriptor { + get { return Descriptor; } + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public SecretMirrorFailureDto() { + OnConstruction(); + } + + partial void OnConstruction(); + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public SecretMirrorFailureDto(SecretMirrorFailureDto other) : this() { + secretName_ = other.secretName_; + message_ = other.message_; + _unknownFields = pb::UnknownFieldSet.Clone(other._unknownFields); + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public SecretMirrorFailureDto Clone() { + return new SecretMirrorFailureDto(this); + } + + /// Field number for the "secret_name" field. + public const int SecretNameFieldNumber = 1; + private string secretName_ = ""; + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public string SecretName { + get { return secretName_; } + set { + secretName_ = pb::ProtoPreconditions.CheckNotNull(value, "value"); + } + } + + /// Field number for the "message" field. + public const int MessageFieldNumber = 2; + private string message_ = ""; + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public string Message { + get { return message_; } + set { + message_ = pb::ProtoPreconditions.CheckNotNull(value, "value"); + } + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public override bool Equals(object other) { + return Equals(other as SecretMirrorFailureDto); + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public bool Equals(SecretMirrorFailureDto other) { + if (ReferenceEquals(other, null)) { + return false; + } + if (ReferenceEquals(other, this)) { + return true; + } + if (SecretName != other.SecretName) return false; + if (Message != other.Message) return false; + return Equals(_unknownFields, other._unknownFields); + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public override int GetHashCode() { + int hash = 1; + if (SecretName.Length != 0) hash ^= SecretName.GetHashCode(); + if (Message.Length != 0) hash ^= Message.GetHashCode(); + if (_unknownFields != null) { + hash ^= _unknownFields.GetHashCode(); + } + return hash; + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public override string ToString() { + return pb::JsonFormatter.ToDiagnosticString(this); + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public void WriteTo(pb::CodedOutputStream output) { + #if !GOOGLE_PROTOBUF_REFSTRUCT_COMPATIBILITY_MODE + output.WriteRawMessage(this); + #else + if (SecretName.Length != 0) { + output.WriteRawTag(10); + output.WriteString(SecretName); + } + if (Message.Length != 0) { + output.WriteRawTag(18); + output.WriteString(Message); + } + if (_unknownFields != null) { + _unknownFields.WriteTo(output); + } + #endif + } + + #if !GOOGLE_PROTOBUF_REFSTRUCT_COMPATIBILITY_MODE + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + void pb::IBufferMessage.InternalWriteTo(ref pb::WriteContext output) { + if (SecretName.Length != 0) { + output.WriteRawTag(10); + output.WriteString(SecretName); + } + if (Message.Length != 0) { + output.WriteRawTag(18); + output.WriteString(Message); + } + if (_unknownFields != null) { + _unknownFields.WriteTo(ref output); + } + } + #endif + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public int CalculateSize() { + int size = 0; + if (SecretName.Length != 0) { + size += 1 + pb::CodedOutputStream.ComputeStringSize(SecretName); + } + if (Message.Length != 0) { + size += 1 + pb::CodedOutputStream.ComputeStringSize(Message); + } + if (_unknownFields != null) { + size += _unknownFields.CalculateSize(); + } + return size; + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public void MergeFrom(SecretMirrorFailureDto other) { + if (other == null) { + return; + } + if (other.SecretName.Length != 0) { + SecretName = other.SecretName; + } + if (other.Message.Length != 0) { + Message = other.Message; + } + _unknownFields = pb::UnknownFieldSet.MergeFrom(_unknownFields, other._unknownFields); + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public void MergeFrom(pb::CodedInputStream input) { + #if !GOOGLE_PROTOBUF_REFSTRUCT_COMPATIBILITY_MODE + input.ReadRawMessage(this); + #else + uint tag; + while ((tag = input.ReadTag()) != 0) { + if ((tag & 7) == 4) { + // Abort on any end group tag. + return; + } + switch(tag) { + default: + _unknownFields = pb::UnknownFieldSet.MergeFieldFrom(_unknownFields, input); + break; + case 10: { + SecretName = input.ReadString(); + break; + } + case 18: { + Message = input.ReadString(); + break; + } + } + } + #endif + } + + #if !GOOGLE_PROTOBUF_REFSTRUCT_COMPATIBILITY_MODE + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + void pb::IBufferMessage.InternalMergeFrom(ref pb::ParseContext input) { + uint tag; + while ((tag = input.ReadTag()) != 0) { + if ((tag & 7) == 4) { + // Abort on any end group tag. + return; + } + switch(tag) { + default: + _unknownFields = pb::UnknownFieldSet.MergeFieldFrom(_unknownFields, ref input); + break; + case 10: { + SecretName = input.ReadString(); + break; + } + case 18: { + Message = input.ReadString(); + break; + } + } + } + } + #endif + + } + + [global::System.Diagnostics.DebuggerDisplayAttribute("{ToString(),nq}")] + public sealed partial class SecretMirrorNodeStatusDto : pb::IMessage + #if !GOOGLE_PROTOBUF_REFSTRUCT_COMPATIBILITY_MODE + , pb::IBufferMessage + #endif + { + private static readonly pb::MessageParser _parser = new pb::MessageParser(() => new SecretMirrorNodeStatusDto()); + private pb::UnknownFieldSet _unknownFields; + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public static pb::MessageParser Parser { get { return _parser; } } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public static pbr::MessageDescriptor Descriptor { + get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CentralControlReflection.Descriptor.MessageTypes[15]; } + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + pbr::MessageDescriptor pb::IMessage.Descriptor { + get { return Descriptor; } + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public SecretMirrorNodeStatusDto() { + OnConstruction(); + } + + partial void OnConstruction(); + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public SecretMirrorNodeStatusDto(SecretMirrorNodeStatusDto other) : this() { + nodeName_ = other.nodeName_; + checkedAtUtc_ = other.checkedAtUtc_ != null ? other.checkedAtUtc_.Clone() : null; + failures_ = other.failures_.Clone(); + _unknownFields = pb::UnknownFieldSet.Clone(other._unknownFields); + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public SecretMirrorNodeStatusDto Clone() { + return new SecretMirrorNodeStatusDto(this); + } + + /// Field number for the "node_name" field. + public const int NodeNameFieldNumber = 1; + private string nodeName_ = ""; + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public string NodeName { + get { return nodeName_; } + set { + nodeName_ = pb::ProtoPreconditions.CheckNotNull(value, "value"); + } + } + + /// Field number for the "checked_at_utc" field. + public const int CheckedAtUtcFieldNumber = 2; + private global::Google.Protobuf.WellKnownTypes.Timestamp checkedAtUtc_; + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public global::Google.Protobuf.WellKnownTypes.Timestamp CheckedAtUtc { + get { return checkedAtUtc_; } + set { + checkedAtUtc_ = value; + } + } + + /// Field number for the "failures" field. + public const int FailuresFieldNumber = 3; + private static readonly pb::FieldCodec _repeated_failures_codec + = pb::FieldCodec.ForMessage(26, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SecretMirrorFailureDto.Parser); + private readonly pbc::RepeatedField failures_ = new pbc::RepeatedField(); + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public pbc::RepeatedField Failures { + get { return failures_; } + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public override bool Equals(object other) { + return Equals(other as SecretMirrorNodeStatusDto); + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public bool Equals(SecretMirrorNodeStatusDto other) { + if (ReferenceEquals(other, null)) { + return false; + } + if (ReferenceEquals(other, this)) { + return true; + } + if (NodeName != other.NodeName) return false; + if (!object.Equals(CheckedAtUtc, other.CheckedAtUtc)) return false; + if(!failures_.Equals(other.failures_)) return false; + return Equals(_unknownFields, other._unknownFields); + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public override int GetHashCode() { + int hash = 1; + if (NodeName.Length != 0) hash ^= NodeName.GetHashCode(); + if (checkedAtUtc_ != null) hash ^= CheckedAtUtc.GetHashCode(); + hash ^= failures_.GetHashCode(); + if (_unknownFields != null) { + hash ^= _unknownFields.GetHashCode(); + } + return hash; + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public override string ToString() { + return pb::JsonFormatter.ToDiagnosticString(this); + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public void WriteTo(pb::CodedOutputStream output) { + #if !GOOGLE_PROTOBUF_REFSTRUCT_COMPATIBILITY_MODE + output.WriteRawMessage(this); + #else + if (NodeName.Length != 0) { + output.WriteRawTag(10); + output.WriteString(NodeName); + } + if (checkedAtUtc_ != null) { + output.WriteRawTag(18); + output.WriteMessage(CheckedAtUtc); + } + failures_.WriteTo(output, _repeated_failures_codec); + if (_unknownFields != null) { + _unknownFields.WriteTo(output); + } + #endif + } + + #if !GOOGLE_PROTOBUF_REFSTRUCT_COMPATIBILITY_MODE + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + void pb::IBufferMessage.InternalWriteTo(ref pb::WriteContext output) { + if (NodeName.Length != 0) { + output.WriteRawTag(10); + output.WriteString(NodeName); + } + if (checkedAtUtc_ != null) { + output.WriteRawTag(18); + output.WriteMessage(CheckedAtUtc); + } + failures_.WriteTo(ref output, _repeated_failures_codec); + if (_unknownFields != null) { + _unknownFields.WriteTo(ref output); + } + } + #endif + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public int CalculateSize() { + int size = 0; + if (NodeName.Length != 0) { + size += 1 + pb::CodedOutputStream.ComputeStringSize(NodeName); + } + if (checkedAtUtc_ != null) { + size += 1 + pb::CodedOutputStream.ComputeMessageSize(CheckedAtUtc); + } + size += failures_.CalculateSize(_repeated_failures_codec); + if (_unknownFields != null) { + size += _unknownFields.CalculateSize(); + } + return size; + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public void MergeFrom(SecretMirrorNodeStatusDto other) { + if (other == null) { + return; + } + if (other.NodeName.Length != 0) { + NodeName = other.NodeName; + } + if (other.checkedAtUtc_ != null) { + if (checkedAtUtc_ == null) { + CheckedAtUtc = new global::Google.Protobuf.WellKnownTypes.Timestamp(); + } + CheckedAtUtc.MergeFrom(other.CheckedAtUtc); + } + failures_.Add(other.failures_); + _unknownFields = pb::UnknownFieldSet.MergeFrom(_unknownFields, other._unknownFields); + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public void MergeFrom(pb::CodedInputStream input) { + #if !GOOGLE_PROTOBUF_REFSTRUCT_COMPATIBILITY_MODE + input.ReadRawMessage(this); + #else + uint tag; + while ((tag = input.ReadTag()) != 0) { + if ((tag & 7) == 4) { + // Abort on any end group tag. + return; + } + switch(tag) { + default: + _unknownFields = pb::UnknownFieldSet.MergeFieldFrom(_unknownFields, input); + break; + case 10: { + NodeName = input.ReadString(); + break; + } + case 18: { + if (checkedAtUtc_ == null) { + CheckedAtUtc = new global::Google.Protobuf.WellKnownTypes.Timestamp(); + } + input.ReadMessage(CheckedAtUtc); + break; + } + case 26: { + failures_.AddEntriesFrom(input, _repeated_failures_codec); + break; + } + } + } + #endif + } + + #if !GOOGLE_PROTOBUF_REFSTRUCT_COMPATIBILITY_MODE + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + void pb::IBufferMessage.InternalMergeFrom(ref pb::ParseContext input) { + uint tag; + while ((tag = input.ReadTag()) != 0) { + if ((tag & 7) == 4) { + // Abort on any end group tag. + return; + } + switch(tag) { + default: + _unknownFields = pb::UnknownFieldSet.MergeFieldFrom(_unknownFields, ref input); + break; + case 10: { + NodeName = input.ReadString(); + break; + } + case 18: { + if (checkedAtUtc_ == null) { + CheckedAtUtc = new global::Google.Protobuf.WellKnownTypes.Timestamp(); + } + input.ReadMessage(CheckedAtUtc); + break; + } + case 26: { + failures_.AddEntriesFrom(ref input, _repeated_failures_codec); + break; + } + } + } + } + #endif + + } + + /// + /// Presence wrapper (same reason as NodeStatusListDto): absent = null ("no data"), + /// present-but-empty = the feature is wired and no node has recorded a status yet. + /// + [global::System.Diagnostics.DebuggerDisplayAttribute("{ToString(),nq}")] + public sealed partial class SecretMirrorStatusListDto : pb::IMessage + #if !GOOGLE_PROTOBUF_REFSTRUCT_COMPATIBILITY_MODE + , pb::IBufferMessage + #endif + { + private static readonly pb::MessageParser _parser = new pb::MessageParser(() => new SecretMirrorStatusListDto()); + private pb::UnknownFieldSet _unknownFields; + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public static pb::MessageParser Parser { get { return _parser; } } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public static pbr::MessageDescriptor Descriptor { + get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CentralControlReflection.Descriptor.MessageTypes[16]; } + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + pbr::MessageDescriptor pb::IMessage.Descriptor { + get { return Descriptor; } + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public SecretMirrorStatusListDto() { + OnConstruction(); + } + + partial void OnConstruction(); + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public SecretMirrorStatusListDto(SecretMirrorStatusListDto other) : this() { + nodes_ = other.nodes_.Clone(); + _unknownFields = pb::UnknownFieldSet.Clone(other._unknownFields); + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public SecretMirrorStatusListDto Clone() { + return new SecretMirrorStatusListDto(this); + } + + /// Field number for the "nodes" field. + public const int NodesFieldNumber = 1; + private static readonly pb::FieldCodec _repeated_nodes_codec + = pb::FieldCodec.ForMessage(10, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SecretMirrorNodeStatusDto.Parser); + private readonly pbc::RepeatedField nodes_ = new pbc::RepeatedField(); + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public pbc::RepeatedField Nodes { + get { return nodes_; } + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public override bool Equals(object other) { + return Equals(other as SecretMirrorStatusListDto); + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public bool Equals(SecretMirrorStatusListDto other) { + if (ReferenceEquals(other, null)) { + return false; + } + if (ReferenceEquals(other, this)) { + return true; + } + if(!nodes_.Equals(other.nodes_)) return false; + return Equals(_unknownFields, other._unknownFields); + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public override int GetHashCode() { + int hash = 1; + hash ^= nodes_.GetHashCode(); + if (_unknownFields != null) { + hash ^= _unknownFields.GetHashCode(); + } + return hash; + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public override string ToString() { + return pb::JsonFormatter.ToDiagnosticString(this); + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public void WriteTo(pb::CodedOutputStream output) { + #if !GOOGLE_PROTOBUF_REFSTRUCT_COMPATIBILITY_MODE + output.WriteRawMessage(this); + #else + nodes_.WriteTo(output, _repeated_nodes_codec); + if (_unknownFields != null) { + _unknownFields.WriteTo(output); + } + #endif + } + + #if !GOOGLE_PROTOBUF_REFSTRUCT_COMPATIBILITY_MODE + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + void pb::IBufferMessage.InternalWriteTo(ref pb::WriteContext output) { + nodes_.WriteTo(ref output, _repeated_nodes_codec); + if (_unknownFields != null) { + _unknownFields.WriteTo(ref output); + } + } + #endif + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public int CalculateSize() { + int size = 0; + size += nodes_.CalculateSize(_repeated_nodes_codec); + if (_unknownFields != null) { + size += _unknownFields.CalculateSize(); + } + return size; + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public void MergeFrom(SecretMirrorStatusListDto other) { + if (other == null) { + return; + } + nodes_.Add(other.nodes_); + _unknownFields = pb::UnknownFieldSet.MergeFrom(_unknownFields, other._unknownFields); + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public void MergeFrom(pb::CodedInputStream input) { + #if !GOOGLE_PROTOBUF_REFSTRUCT_COMPATIBILITY_MODE + input.ReadRawMessage(this); + #else + uint tag; + while ((tag = input.ReadTag()) != 0) { + if ((tag & 7) == 4) { + // Abort on any end group tag. + return; + } + switch(tag) { + default: + _unknownFields = pb::UnknownFieldSet.MergeFieldFrom(_unknownFields, input); + break; + case 10: { + nodes_.AddEntriesFrom(input, _repeated_nodes_codec); + break; + } + } + } + #endif + } + + #if !GOOGLE_PROTOBUF_REFSTRUCT_COMPATIBILITY_MODE + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + void pb::IBufferMessage.InternalMergeFrom(ref pb::ParseContext input) { + uint tag; + while ((tag = input.ReadTag()) != 0) { + if ((tag & 7) == 4) { + // Abort on any end group tag. + return; + } + switch(tag) { + default: + _unknownFields = pb::UnknownFieldSet.MergeFieldFrom(_unknownFields, ref input); + break; + case 10: { + nodes_.AddEntriesFrom(ref input, _repeated_nodes_codec); + break; + } + } + } + } + #endif + + } + /// /// Site -> Central: periodic site health report. Mirrors Commons SiteHealthReport. /// Additive-only evolution: field numbers are never reused. @@ -4222,7 +4934,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { [global::System.Diagnostics.DebuggerNonUserCodeAttribute] [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public static pbr::MessageDescriptor Descriptor { - get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CentralControlReflection.Descriptor.MessageTypes[14]; } + get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CentralControlReflection.Descriptor.MessageTypes[17]; } } [global::System.Diagnostics.DebuggerNonUserCodeAttribute] @@ -4274,6 +4986,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { detachedScriptThreads_ = other.detachedScriptThreads_; scriptRunShedCount_ = other.scriptRunShedCount_; siteStreamAlarmDropCount_ = other.siteStreamAlarmDropCount_; + secretMirrorStatuses_ = other.secretMirrorStatuses_ != null ? other.secretMirrorStatuses_.Clone() : null; _unknownFields = pb::UnknownFieldSet.Clone(other._unknownFields); } @@ -4721,6 +5434,23 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { } } + /// Field number for the "secret_mirror_statuses" field. + public const int SecretMirrorStatusesFieldNumber = 33; + private global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SecretMirrorStatusListDto secretMirrorStatuses_; + /// + /// Review finding M1: every site node's deployed-secrets mirror status (the standby's + /// included — it is read from a LocalDb-replicated table on the active node). + /// Absent means null ("no data": central self-report, pre-M1 site, or not read yet). + /// + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SecretMirrorStatusListDto SecretMirrorStatuses { + get { return secretMirrorStatuses_; } + set { + secretMirrorStatuses_ = value; + } + } + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public override bool Equals(object other) { @@ -4768,6 +5498,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { if (DetachedScriptThreads != other.DetachedScriptThreads) return false; if (ScriptRunShedCount != other.ScriptRunShedCount) return false; if (SiteStreamAlarmDropCount != other.SiteStreamAlarmDropCount) return false; + if (!object.Equals(SecretMirrorStatuses, other.SecretMirrorStatuses)) return false; return Equals(_unknownFields, other._unknownFields); } @@ -4807,6 +5538,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { if (DetachedScriptThreads != 0) hash ^= DetachedScriptThreads.GetHashCode(); if (ScriptRunShedCount != 0) hash ^= ScriptRunShedCount.GetHashCode(); if (SiteStreamAlarmDropCount != 0L) hash ^= SiteStreamAlarmDropCount.GetHashCode(); + if (secretMirrorStatuses_ != null) hash ^= SecretMirrorStatuses.GetHashCode(); if (_unknownFields != null) { hash ^= _unknownFields.GetHashCode(); } @@ -4939,6 +5671,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { output.WriteRawTag(128, 2); output.WriteInt64(SiteStreamAlarmDropCount); } + if (secretMirrorStatuses_ != null) { + output.WriteRawTag(138, 2); + output.WriteMessage(SecretMirrorStatuses); + } if (_unknownFields != null) { _unknownFields.WriteTo(output); } @@ -5063,6 +5799,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { output.WriteRawTag(128, 2); output.WriteInt64(SiteStreamAlarmDropCount); } + if (secretMirrorStatuses_ != null) { + output.WriteRawTag(138, 2); + output.WriteMessage(SecretMirrorStatuses); + } if (_unknownFields != null) { _unknownFields.WriteTo(ref output); } @@ -5163,6 +5903,9 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { if (SiteStreamAlarmDropCount != 0L) { size += 2 + pb::CodedOutputStream.ComputeInt64Size(SiteStreamAlarmDropCount); } + if (secretMirrorStatuses_ != null) { + size += 2 + pb::CodedOutputStream.ComputeMessageSize(SecretMirrorStatuses); + } if (_unknownFields != null) { size += _unknownFields.CalculateSize(); } @@ -5290,6 +6033,12 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { if (other.SiteStreamAlarmDropCount != 0L) { SiteStreamAlarmDropCount = other.SiteStreamAlarmDropCount; } + if (other.secretMirrorStatuses_ != null) { + if (secretMirrorStatuses_ == null) { + SecretMirrorStatuses = new global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SecretMirrorStatusListDto(); + } + SecretMirrorStatuses.MergeFrom(other.SecretMirrorStatuses); + } _unknownFields = pb::UnknownFieldSet.MergeFrom(_unknownFields, other._unknownFields); } @@ -5467,6 +6216,13 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { SiteStreamAlarmDropCount = input.ReadInt64(); break; } + case 266: { + if (secretMirrorStatuses_ == null) { + SecretMirrorStatuses = new global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SecretMirrorStatusListDto(); + } + input.ReadMessage(SecretMirrorStatuses); + break; + } } } #endif @@ -5644,6 +6400,13 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { SiteStreamAlarmDropCount = input.ReadInt64(); break; } + case 266: { + if (secretMirrorStatuses_ == null) { + SecretMirrorStatuses = new global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SecretMirrorStatusListDto(); + } + input.ReadMessage(SecretMirrorStatuses); + break; + } } } } @@ -5669,7 +6432,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { [global::System.Diagnostics.DebuggerNonUserCodeAttribute] [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public static pbr::MessageDescriptor Descriptor { - get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CentralControlReflection.Descriptor.MessageTypes[15]; } + get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CentralControlReflection.Descriptor.MessageTypes[18]; } } [global::System.Diagnostics.DebuggerNonUserCodeAttribute] @@ -5984,7 +6747,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { [global::System.Diagnostics.DebuggerNonUserCodeAttribute] [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public static pbr::MessageDescriptor Descriptor { - get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CentralControlReflection.Descriptor.MessageTypes[16]; } + get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CentralControlReflection.Descriptor.MessageTypes[19]; } } [global::System.Diagnostics.DebuggerNonUserCodeAttribute] diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/CentralControlDtoMapper.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/CentralControlDtoMapper.cs index 0ffc5fd2..cf977609 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/CentralControlDtoMapper.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/CentralControlDtoMapper.cs @@ -559,6 +559,32 @@ public static class CentralControlDtoMapper dto.SiteAuditBacklog = snapshot; } + // Review finding M1: wrapper presence keeps null ("no data") distinct from empty. + if (msg.SecretMirrorStatuses is { } mirrorStatuses) + { + var wrapper = new SecretMirrorStatusListDto(); + foreach (var node in mirrorStatuses) + { + var nodeDto = new SecretMirrorNodeStatusDto + { + NodeName = node.NodeName, + CheckedAtUtc = Timestamp.FromDateTimeOffset(node.CheckedAtUtc), + }; + foreach (var failure in node.Failures) + { + nodeDto.Failures.Add(new SecretMirrorFailureDto + { + SecretName = failure.SecretName, + Message = failure.Message, + }); + } + + wrapper.Nodes.Add(nodeDto); + } + + dto.SecretMirrorStatuses = wrapper; + } + return dto; } @@ -620,6 +646,19 @@ public static class CentralControlDtoMapper OnDiskBytes: snapshot.OnDiskBytes); } + List? mirrorStatuses = null; + if (dto.SecretMirrorStatuses is { } mirrorWrapper) + { + mirrorStatuses = new List(mirrorWrapper.Nodes.Count); + foreach (var node in mirrorWrapper.Nodes) + { + mirrorStatuses.Add(new SecretMirrorNodeStatus( + node.NodeName, + node.CheckedAtUtc?.ToDateTimeOffset() ?? DateTimeOffset.MinValue, + node.Failures.Select(f => new SecretMirrorFailure(f.SecretName, f.Message)).ToList())); + } + } + return new SiteHealthReport( SiteId: dto.SiteId, SequenceNumber: dto.SequenceNumber, @@ -657,6 +696,7 @@ public static class CentralControlDtoMapper DetachedScriptThreads = dto.DetachedScriptThreads, ScriptRunShedCount = dto.ScriptRunShedCount, SiteStreamAlarmDropCount = dto.SiteStreamAlarmDropCount, + SecretMirrorStatuses = mirrorStatuses, }; } diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Protos/central_control.proto b/src/ZB.MOM.WW.ScadaBridge.Communication/Protos/central_control.proto index 29014532..e092b3f9 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Protos/central_control.proto +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Protos/central_control.proto @@ -193,6 +193,25 @@ message NodeStatusListDto { repeated NodeStatusDto nodes = 1; } +// Review finding M1: one site node's deployed-secrets mirror failures. Names and +// human-readable messages only — never a secret value or ciphertext. +message SecretMirrorFailureDto { + string secret_name = 1; + string message = 2; +} + +message SecretMirrorNodeStatusDto { + string node_name = 1; + google.protobuf.Timestamp checked_at_utc = 2; + repeated SecretMirrorFailureDto failures = 3; +} + +// Presence wrapper (same reason as NodeStatusListDto): absent = null ("no data"), +// present-but-empty = the feature is wired and no node has recorded a status yet. +message SecretMirrorStatusListDto { + repeated SecretMirrorNodeStatusDto nodes = 1; +} + // Site -> Central: periodic site health report. Mirrors Commons SiteHealthReport. // Additive-only evolution: field numbers are never reused. message SiteHealthReportDto { @@ -245,6 +264,10 @@ message SiteHealthReportDto { // dedicated alarm publish queue because it was at capacity. Point-in-time // (not reset on collect), like site_event_log_write_failures. int64 site_stream_alarm_drop_count = 32; + // Review finding M1: every site node's deployed-secrets mirror status (the standby's + // included — it is read from a LocalDb-replicated table on the active node). + // Absent means null ("no data": central self-report, pre-M1 site, or not read yet). + SecretMirrorStatusListDto secret_mirror_statuses = 33; } // Central -> Site: health report ack, so a lost report is observable. diff --git a/src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ISiteHealthCollector.cs b/src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ISiteHealthCollector.cs index 0381f923..a1bdac40 100644 --- a/src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ISiteHealthCollector.cs +++ b/src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ISiteHealthCollector.cs @@ -198,6 +198,19 @@ public interface ISiteHealthCollector // Default no-op so test fakes do not need to be updated. } + /// + /// Replace the latest per-node deployed-secrets mirror statuses (review finding M1) used + /// by the SecretMirrorStatuses field of the next call. + /// Refreshed periodically by the site-side SecretMirrorStatusReporter from the + /// LocalDb-replicated secret_mirror_status table, so the ACTIVE node's report + /// carries the STANDBY node's mirror failures too. Point-in-time: NOT reset on collect. + /// + /// Every node's row, or for "no data". + void SetSecretMirrorStatuses(IReadOnlyList? statuses) + { + // Default no-op so test fakes do not need to be updated. + } + /// /// Replace the latest script-execution-scheduler gauges (queue depth, busy /// thread count, and age in seconds of the oldest in-flight script) used by diff --git a/src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthCollector.cs b/src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthCollector.cs index 200a4e8a..da08d5ac 100644 --- a/src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthCollector.cs +++ b/src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthCollector.cs @@ -32,6 +32,10 @@ public class SiteHealthCollector : ISiteHealthCollector // -1 = reconciler SKIPPED (no IDeploymentConfigFetcher) // Null must stay distinct from -1: "we don't know" is not "self-heal is missing". private volatile int _startupReconcilerActive; + // Review finding M1: every site node's deployed-secrets mirror status, refreshed by the + // SiteRuntime SecretMirrorStatusReporter. Null = the reporter has not run (or its read + // failed before it ever succeeded) — "no data", distinct from an empty list. + private volatile IReadOnlyList? _secretMirrorStatuses; private readonly ConcurrentDictionary _connectionStatuses = new(); private readonly ConcurrentDictionary _tagResolutionCounts = new(); private readonly ConcurrentDictionary _connectionEndpoints = new(); @@ -130,6 +134,12 @@ public class SiteHealthCollector : ISiteHealthCollector _startupReconcilerActive = active ? 1 : -1; } + /// + public void SetSecretMirrorStatuses(IReadOnlyList? statuses) + { + _secretMirrorStatuses = statuses?.ToList(); + } + /// public void UpdateConnectionHealth(string connectionName, ConnectionHealth health) { @@ -333,7 +343,9 @@ public class SiteHealthCollector : ISiteHealthCollector SiteStreamAlarmDropCount = Interlocked.Read(ref _siteStreamAlarmDropCount), // Tri-state: 0 stays null ("not reported"), which must never collapse into // false ("self-heal is missing"). Register row 73. - StartupReconcilerActive = startupReconciler == 0 ? null : startupReconciler > 0 + StartupReconcilerActive = startupReconciler == 0 ? null : startupReconciler > 0, + // M1: point-in-time, not drained; null stays null ("no data"). + SecretMirrorStatuses = _secretMirrorStatuses }; } } diff --git a/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs b/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs index 85231b19..6d27556d 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs @@ -48,7 +48,7 @@ namespace ZB.MOM.WW.ScadaBridge.Host; public static class SiteLocalDbSetup { /// - /// The eleven tables that participate in replication — registered when this node has a peer, + /// The twelve tables that participate in replication — registered when this node has a peer, /// and deregistered (stale triggers dropped) when it does not. /// /// @@ -69,6 +69,13 @@ public static class SiteLocalDbSetup /// deployed_secrets row, and replicating the ledger would make one node's progress /// masquerade as the other's. /// + /// + /// secret_mirror_status is the twelfth entry (review finding M1). It is per-node + /// DATA — one row per node, keyed by node_name, each written only by its own node — + /// but it is replicated precisely so the active node can read the standby's row and put + /// it on the site health report (only the active node sends one). Explicit primary key, + /// no FK, no BLOB columns. + /// /// internal static readonly string[] ReplicatedTables = [ @@ -83,6 +90,7 @@ public static class SiteLocalDbSetup "data_connection_definitions", "native_alarm_state", "deployed_secrets", + "secret_mirror_status", ]; /// @@ -140,8 +148,9 @@ public static class SiteLocalDbSetup // ELEVENTH entry to ReplicatedTables on an already-replicating site streams a // snapshot of only that one table, where through 0.2.x the flag was a single // per-database bit and the same edit would have re-streamed all eleven in full, in - // both directions. Nothing about the first boot changes: enabling replication on a - // site that has been running without it seeds all eleven at once, the flagged set is + // both directions (the same holds for secret_mirror_status, the twelfth, added by + // review finding M1). Nothing about the first boot changes: enabling replication on a + // site that has been running without it seeds all twelve at once, the flagged set is // therefore every registered table, and the library sends that as an ordinary full // snapshot. A hypothetical "register the whole list in one call" API would silently // give the coarse behaviour back. diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageSchema.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageSchema.cs index 66532b78..c4793233 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageSchema.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageSchema.cs @@ -3,8 +3,8 @@ using Microsoft.Data.Sqlite; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; /// -/// DDL for the site's eleven configuration tables (nine original plus -/// deployed_secrets and mirrored_secrets), extracted from +/// DDL for the site's twelve configuration tables (nine original plus +/// deployed_secrets, mirrored_secrets and secret_mirror_status), extracted from /// so it can be applied by whoever owns the database /// file. /// @@ -23,7 +23,7 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; public static class SiteStorageSchema { /// - /// Creates the eleven site configuration tables when absent, and additively upgrades + /// Creates the twelve site configuration tables when absent, and additively upgrades /// tables created by an older build. Idempotent — safe to run on every startup. /// /// An open connection to the database that should hold the tables. @@ -136,6 +136,20 @@ public static class SiteStorageSchema revision INTEGER NOT NULL, mirrored_at TEXT NOT NULL ); + + -- Per-node deployed-secrets mirror status (review finding M1): one row per site + -- node, keyed by node name, holding that node's CURRENT mirror failures as a JSON + -- array of {SecretName, Message} (names and messages only — never values or + -- ciphertext). Unlike mirrored_secrets this IS replicated + -- (SiteLocalDbSetup.ReplicatedTables): each node writes only its own row, and the + -- replication is exactly what lets the ACTIVE node's health report carry the + -- STANDBY node's failures to central. Written only when a node's failure set + -- changes (plus once per start-up), never per mirror tick. No foreign keys. + CREATE TABLE IF NOT EXISTS secret_mirror_status ( + node_name TEXT PRIMARY KEY, + failures_json TEXT NOT NULL, + checked_at TEXT NOT NULL + ); "; command.ExecuteNonQuery(); } diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs index 880c0cbf..9ae4e132 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs @@ -3,6 +3,7 @@ using Microsoft.Data.Sqlite; using Microsoft.Extensions.Logging; using ZB.MOM.WW.LocalDb; using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts; +using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; @@ -924,6 +925,87 @@ public class SiteStorageService await command.ExecuteNonQueryAsync(); } + // ── Secret mirror status (review finding M1 — LocalDb-replicated, one row per node) ── + + /// + /// Records 's current deployed-secrets mirror failures in the + /// replicated secret_mirror_status table (UPSERT on the node name). An empty + /// list records a healthy mirror. Every call is a replicated + /// write, so callers write only when the failure set changes. + /// + /// This node's name — the row key. + /// The node's current failures (names and messages only, never values). + /// When the recorded failure set was observed. + /// A task that completes when the row has been stored. + public async Task SetSecretMirrorStatusAsync( + string nodeName, IReadOnlyList failures, DateTimeOffset checkedAtUtc) + { + ArgumentException.ThrowIfNullOrWhiteSpace(nodeName); + ArgumentNullException.ThrowIfNull(failures); + + await using var connection = OpenConnection(); + + await using var command = connection.CreateCommand(); + command.CommandText = @" + INSERT INTO secret_mirror_status (node_name, failures_json, checked_at) + VALUES (@nodeName, @failures, @checkedAt) + ON CONFLICT(node_name) DO UPDATE SET + failures_json = excluded.failures_json, + checked_at = excluded.checked_at"; + + command.Parameters.AddWithValue("@nodeName", nodeName); + command.Parameters.AddWithValue("@failures", JsonSerializer.Serialize(failures)); + command.Parameters.AddWithValue("@checkedAt", checkedAtUtc.ToUniversalTime().ToString("O")); + + await command.ExecuteNonQueryAsync(); + } + + /// + /// Returns every node's row from secret_mirror_status, ordered by node name. A row + /// whose failures_json cannot be parsed is returned with a single synthetic failure + /// saying so, rather than failing the whole read or silently reading as healthy. + /// + /// A task that resolves to every node's recorded mirror status. + public async Task> GetSecretMirrorStatusesAsync() + { + await using var connection = OpenConnection(); + + await using var command = connection.CreateCommand(); + command.CommandText = + "SELECT node_name, failures_json, checked_at FROM secret_mirror_status ORDER BY node_name"; + + var results = new List(); + await using var reader = await command.ExecuteReaderAsync(); + while (await reader.ReadAsync()) + { + var nodeName = reader.GetString(0); + IReadOnlyList failures; + try + { + failures = JsonSerializer.Deserialize>(reader.GetString(1)) + ?? []; + } + catch (Exception ex) when (ex is JsonException or NotSupportedException) + { + failures = + [ + new SecretMirrorFailure( + "(unreadable)", + $"Mirror status row for node '{nodeName}' could not be parsed ({ex.GetType().Name})"), + ]; + } + + var checkedAt = DateTimeOffset.TryParse( + reader.GetString(2), null, System.Globalization.DateTimeStyles.RoundtripKind, out var parsed) + ? parsed + : DateTimeOffset.MinValue; + + results.Add(new SecretMirrorNodeStatus(nodeName, checkedAt, failures)); + } + + return results; + } + // ── Shared Script CRUD ── /// diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Secrets/DeployedSecretsMirror.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Secrets/DeployedSecretsMirror.cs index 993d1565..b07280df 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Secrets/DeployedSecretsMirror.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Secrets/DeployedSecretsMirror.cs @@ -1,7 +1,9 @@ using Microsoft.Extensions.Hosting; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; +using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services; using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts; +using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.Secrets.Abstractions; @@ -153,6 +155,18 @@ public sealed record MirrorResult( /// so a LATER failure — even one with an identical fingerprint — logs at Error again. This state /// is in-memory only (per node process), which matches the ledger's own node-local scope. /// +/// +/// Status row for central (review finding M1). A failure on the STANDBY node used to be +/// visible only in that node's own log — site health reports are sent by the active node only. +/// After each pass this mirror records its node's current failure set (secret name + message, +/// never a value) in the LocalDb-replicated secret_mirror_status table, keyed by node +/// name; SecretMirrorStatusReporter on the active node reads every node's row onto the +/// health report. Because every write replicates via CDC, the row is written only when the +/// failure set (compared as sorted name+message pairs) differs from what this process last +/// wrote — plus once on the first pass after start-up, so a restart refreshes a stale row. The +/// write is best-effort: a failure logs a Warning (repeats at Debug) and is retried on the next +/// pass, and never affects the pass's own . +/// /// public sealed class DeployedSecretsMirror : BackgroundService { @@ -165,6 +179,17 @@ public sealed class DeployedSecretsMirror : BackgroundService private readonly IOptions _options; private readonly ILogger _logger; private readonly TimeProvider _time; + private readonly string _nodeName; + + /// + /// The failure set (sorted name+message pairs) this process last wrote successfully to + /// secret_mirror_status, or before the first successful write + /// — which is what forces the once-per-start-up write (review finding M1). + /// + private IReadOnlyList? _lastWrittenStatus; + + /// Whether a failed status write has already been logged at Warning (de-spam). + private bool _statusWriteFailureLogged; private readonly SemaphoreSlim _gate = new(1, 1); private bool _disposed; @@ -199,6 +224,11 @@ public sealed class DeployedSecretsMirror : BackgroundService /// Supplies . /// Logger instance. Never logs secret values — names, revisions, and KEK ids only. /// Clock driving the reconcile loop's and the tombstone-recovery stamp. + /// + /// Supplies this node's name (node-a/node-b), the key of its + /// secret_mirror_status row. Optional: when absent or unconfigured, the machine name + /// is used instead. + /// public DeployedSecretsMirror( SiteStorageService storage, ISecretStore store, @@ -206,7 +236,8 @@ public sealed class DeployedSecretsMirror : BackgroundService ISecretCacheInvalidator? invalidator, IOptions options, ILogger logger, - TimeProvider time) + TimeProvider time, + INodeIdentityProvider? nodeIdentity = null) { _storage = storage ?? throw new ArgumentNullException(nameof(storage)); _store = store ?? throw new ArgumentNullException(nameof(store)); @@ -215,6 +246,7 @@ public sealed class DeployedSecretsMirror : BackgroundService _options = options ?? throw new ArgumentNullException(nameof(options)); _logger = logger ?? throw new ArgumentNullException(nameof(logger)); _time = time ?? throw new ArgumentNullException(nameof(time)); + _nodeName = nodeIdentity?.NodeName ?? Environment.MachineName; } /// @@ -314,6 +346,8 @@ public sealed class DeployedSecretsMirror : BackgroundService rows.Count(r => !r.IsRemoved) + unreadable.Count(u => !u.IsRemoved), failed.Count); } + await RecordStatusIfChangedAsync(failedNames, failed).ConfigureAwait(false); + return result; } finally @@ -322,6 +356,42 @@ public sealed class DeployedSecretsMirror : BackgroundService } } + /// + /// Writes this node's secret_mirror_status row when the pass's failure set differs + /// from the last one this process wrote (or nothing has been written yet since start-up). + /// Best-effort: never throws, so the pass result is unaffected by a status-write fault. + /// See the class remarks (review finding M1). + /// + private async Task RecordStatusIfChangedAsync(IReadOnlyList failedNames, IReadOnlyList failed) + { + var current = failedNames + .Select((name, i) => new SecretMirrorFailure(name, failed[i])) + .OrderBy(f => f.SecretName, StringComparer.Ordinal) + .ThenBy(f => f.Message, StringComparer.Ordinal) + .ToList(); + + if (_lastWrittenStatus is not null && _lastWrittenStatus.SequenceEqual(current)) + { + return; + } + + try + { + await _storage.SetSecretMirrorStatusAsync(_nodeName, current, _time.GetUtcNow()).ConfigureAwait(false); + _lastWrittenStatus = current; + _statusWriteFailureLogged = false; + } + catch (Exception ex) + { + _logger.Log( + _statusWriteFailureLogged ? LogLevel.Debug : LogLevel.Warning, + ex, + "Failed to record the deployed secrets mirror status for node {NodeName}; retrying next pass", + _nodeName); + _statusWriteFailureLogged = true; + } + } + /// /// Handles a tombstoned deployed_secrets row: deletes it from this node's store, but /// only if this node had previously ledgered it as mirrored — an out-of-band, same-named diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Secrets/SecretMirrorStatusReporter.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Secrets/SecretMirrorStatusReporter.cs new file mode 100644 index 00000000..85abb4d8 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Secrets/SecretMirrorStatusReporter.cs @@ -0,0 +1,95 @@ +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; +using ZB.MOM.WW.ScadaBridge.HealthMonitoring; +using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; + +namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Secrets; + +/// +/// Review finding M1: site-side hosted service that periodically reads EVERY node's row from +/// the LocalDb-replicated secret_mirror_status table and pushes the list into +/// , so the next health report +/// carries each site node's deployed-secrets mirror failures — the standby's included. +/// +/// +/// +/// Runs on every site node (the read is a cheap local SELECT), but only the active node's +/// report is ever sent (HealthReportSender), which is exactly why the status lives in a +/// REPLICATED table: the standby writes its row, CDC carries it to the active node, and this +/// reporter on the active node forwards it. A site pair WITHOUT LocalDb replication reports +/// only the active node's row — the same prerequisite the deployed secrets themselves have. +/// +/// +/// Lives in SiteRuntime (which already references HealthMonitoring and owns +/// ), following SiteStreamAlarmDropReporter: immediate +/// first probe, fixed cadence, exceptions logged and swallowed. A failed read leaves the +/// previous snapshot on the collector rather than clearing it. +/// +/// +public sealed class SecretMirrorStatusReporter : BackgroundService +{ + /// Default poll cadence (30 s), matching the site health report interval. + internal static readonly TimeSpan DefaultPollInterval = TimeSpan.FromSeconds(30); + + private readonly ISiteHealthCollector _collector; + private readonly SiteStorageService _storage; + private readonly ILogger _logger; + private readonly TimeSpan _pollInterval; + + /// Initializes a new instance of . + /// The site health collector receiving the statuses. + /// Site storage exposing secret_mirror_status. + /// Logger instance. + /// Poll interval override; defaults to (30 s). + public SecretMirrorStatusReporter( + ISiteHealthCollector collector, + SiteStorageService storage, + ILogger logger, + TimeSpan? pollInterval = null) + { + _collector = collector ?? throw new ArgumentNullException(nameof(collector)); + _storage = storage ?? throw new ArgumentNullException(nameof(storage)); + _logger = logger ?? throw new ArgumentNullException(nameof(logger)); + _pollInterval = pollInterval ?? DefaultPollInterval; + } + + /// + protected override async Task ExecuteAsync(CancellationToken stoppingToken) + { + await ProbeAsync().ConfigureAwait(false); + + while (!stoppingToken.IsCancellationRequested) + { + try + { + await Task.Delay(_pollInterval, stoppingToken).ConfigureAwait(false); + } + catch (OperationCanceledException) + { + break; + } + + await ProbeAsync().ConfigureAwait(false); + } + } + + /// + /// Reads every node's mirror status and pushes it onto the collector; never throws. + /// Public so a test can drive one deterministic probe. + /// + /// A task that completes when the probe has finished. + public async Task ProbeAsync() + { + try + { + var statuses = await _storage.GetSecretMirrorStatusesAsync().ConfigureAwait(false); + _collector.SetSecretMirrorStatuses(statuses); + } + catch (Exception ex) + { + // Catch-all is deliberate: health reporting must never take the node down, and + // the loop must survive so the next tick retries. + _logger.LogWarning(ex, "SecretMirrorStatusReporter probe failed; next tick will retry."); + } + } +} diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs index bbb8b139..a8bffc1b 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs @@ -140,6 +140,13 @@ public static class ServiceCollectionExtensions services.AddSingleton(); services.AddHostedService(sp => sp.GetRequiredService()); + // Review finding M1: forward every node's replicated secret_mirror_status row onto the + // site health report, so the ACTIVE node's report carries the STANDBY's mirror failures. + services.AddHostedService(sp => new SecretMirrorStatusReporter( + sp.GetRequiredService(), + sp.GetRequiredService(), + sp.GetRequiredService>())); + return services; } diff --git a/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Pages/HealthPageTests.cs b/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Pages/HealthPageTests.cs index d90fe937..f141eb2c 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Pages/HealthPageTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Pages/HealthPageTests.cs @@ -559,6 +559,70 @@ public class HealthPageTests : BunitContext StartupReconcilerActive = active, }; + // ----- Review finding M1: per-node deployed-secrets mirror failures ------------------ + // + // The standby's failures ride the active node's report; the card must name the node, + // the secret and the message. Null ("no data") and all-healthy nodes render nothing. + + [Fact] + public void Renders_SecretMirrorFailures_LabelledByNode() + { + SeedStates(new SiteHealthState + { + SiteId = "site-a", + IsOnline = true, + LastHeartbeatAt = DateTimeOffset.UtcNow, + LatestReport = ReportWithReconciler("site-a", active: true) with + { + SecretMirrorStatuses = + [ + new SecretMirrorNodeStatus("node-a", DateTimeOffset.UtcNow, []), + new SecretMirrorNodeStatus( + "node-b", + DateTimeOffset.UtcNow, + [new SecretMirrorFailure("OPC-PWD", "did not decrypt on this node")]), + ], + }, + }); + + var cut = RenderHealthPage(); + + cut.WaitForAssertion(() => + { + Assert.Contains("data-test=\"secret-mirror-badge\"", cut.Markup); + var panel = cut.Find("[data-test=secret-mirror-failures]"); + Assert.Contains("node-b", panel.TextContent); + Assert.Contains("OPC-PWD", panel.TextContent); + Assert.Contains("did not decrypt on this node", panel.TextContent); + Assert.DoesNotContain("node-a", panel.TextContent); + }); + } + + [Theory] + [InlineData(false)] // null = "no data" (central self-report, pre-M1 site) + [InlineData(true)] // every node healthy + public void DoesNotRender_SecretMirrorWarning_WhenNoNodeIsFailing(bool reported) + { + SeedStates(new SiteHealthState + { + SiteId = "site-a", + IsOnline = true, + LastHeartbeatAt = DateTimeOffset.UtcNow, + LatestReport = ReportWithReconciler("site-a", active: true) with + { + SecretMirrorStatuses = reported + ? [new SecretMirrorNodeStatus("node-a", DateTimeOffset.UtcNow, [])] + : null, + }, + }); + + var cut = RenderHealthPage(); + + cut.WaitForAssertion(() => Assert.Contains("site-a", cut.Markup)); + Assert.DoesNotContain("data-test=\"secret-mirror-badge\"", cut.Markup); + Assert.DoesNotContain("data-test=\"secret-mirror-failures\"", cut.Markup); + } + [Fact] public void Renders_OfflineSince_ForOfflineSiteWithStatusChange() { diff --git a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/CentralControlDtoMapperTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/CentralControlDtoMapperTests.cs index 31d83c0e..cdabecb9 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/CentralControlDtoMapperTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/CentralControlDtoMapperTests.cs @@ -369,6 +369,7 @@ public class CentralControlDtoMapperTests Assert.Null(roundTripped.LocalDbReplicationConnected); Assert.Null(roundTripped.LocalDbOplogBacklog); Assert.Null(roundTripped.StartupReconcilerActive); + Assert.Null(roundTripped.SecretMirrorStatuses); } [Fact] @@ -384,10 +385,14 @@ public class CentralControlDtoMapperTests DataConnectionEndpoints = new Dictionary(), DataConnectionTagQuality = new Dictionary(), ClusterNodes = [], + SecretMirrorStatuses = [], }; var roundTripped = CentralControlDtoMapper.FromDto(Wire(CentralControlDtoMapper.ToDto(original))); + Assert.NotNull(roundTripped.SecretMirrorStatuses); + Assert.Empty(roundTripped.SecretMirrorStatuses); + Assert.NotNull(roundTripped.DataConnectionEndpoints); Assert.Empty(roundTripped.DataConnectionEndpoints); Assert.NotNull(roundTripped.DataConnectionTagQuality); @@ -559,6 +564,18 @@ public class CentralControlDtoMapperTests DetachedScriptThreads = 2, ScriptRunShedCount = 13, SiteStreamAlarmDropCount = 41L, + // Review finding M1: every node's secret mirror status, standby included. + SecretMirrorStatuses = + [ + new SecretMirrorNodeStatus("node-a", SampleInstant, []), + new SecretMirrorNodeStatus( + "node-b", + SampleInstant.AddMinutes(-3), + [ + new SecretMirrorFailure("OPC-PWD", "did not decrypt on this node"), + new SecretMirrorFailure("MX-KEY", "could not be confirmed applied"), + ]), + ], }; /// @@ -611,6 +628,17 @@ public class CentralControlDtoMapperTests Assert.Equal(expected.DetachedScriptThreads, actual.DetachedScriptThreads); Assert.Equal(expected.ScriptRunShedCount, actual.ScriptRunShedCount); Assert.Equal(expected.SiteStreamAlarmDropCount, actual.SiteStreamAlarmDropCount); + Assert.Equal(expected.SecretMirrorStatuses is null, actual.SecretMirrorStatuses is null); + if (expected.SecretMirrorStatuses is not null) + { + Assert.Equal(expected.SecretMirrorStatuses.Count, actual.SecretMirrorStatuses!.Count); + for (var i = 0; i < expected.SecretMirrorStatuses.Count; i++) + { + Assert.Equal(expected.SecretMirrorStatuses[i].NodeName, actual.SecretMirrorStatuses[i].NodeName); + Assert.Equal(expected.SecretMirrorStatuses[i].CheckedAtUtc, actual.SecretMirrorStatuses[i].CheckedAtUtc); + Assert.Equal(expected.SecretMirrorStatuses[i].Failures, actual.SecretMirrorStatuses[i].Failures); + } + } } /// diff --git a/tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/SecretMirrorStatusHealthFieldTests.cs b/tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/SecretMirrorStatusHealthFieldTests.cs new file mode 100644 index 00000000..0f7ac2ea --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/SecretMirrorStatusHealthFieldTests.cs @@ -0,0 +1,45 @@ +using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health; + +namespace ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests; + +/// +/// Review finding M1 — SiteHealthReport.SecretMirrorStatuses carries every site node's +/// deployed-secrets mirror failures (the standby's included) on the active node's report. +/// +public class SecretMirrorStatusHealthFieldTests +{ + [Fact] + public void CollectReport_NeverTold_LeavesTheFieldNull() + { + Assert.Null(new SiteHealthCollector().CollectReport("TestSite").SecretMirrorStatuses); + } + + [Fact] + public void CollectReport_CarriesTheLatestSnapshot_AndDoesNotDrainIt() + { + var collector = new SiteHealthCollector(); + collector.SetSecretMirrorStatuses( + [ + new SecretMirrorNodeStatus("node-a", DateTimeOffset.UtcNow, []), + new SecretMirrorNodeStatus( + "node-b", DateTimeOffset.UtcNow, [new SecretMirrorFailure("S1", "did not decrypt")]), + ]); + + var first = collector.CollectReport("TestSite").SecretMirrorStatuses!; + var second = collector.CollectReport("TestSite").SecretMirrorStatuses!; + + Assert.Equal(["node-a", "node-b"], first.Select(s => s.NodeName)); + Assert.Equal(2, second.Count); // point-in-time, not interval-drained + Assert.Equal("S1", Assert.Single(first[1].Failures).SecretName); + } + + [Fact] + public void SetNull_ResetsToNoData() + { + var collector = new SiteHealthCollector(); + collector.SetSecretMirrorStatuses([]); + collector.SetSecretMirrorStatuses(null); + + Assert.Null(collector.CollectReport("TestSite").SecretMirrorStatuses); + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbCdcRegistrationTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbCdcRegistrationTests.cs index 36fd3007..a2469924 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbCdcRegistrationTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbCdcRegistrationTests.cs @@ -212,7 +212,7 @@ public class SiteLocalDbCdcRegistrationTests : IDisposable // ---- helpers ---------------------------------------------------------------------- /// - /// The eleven replicated tables, plus the two deliberately-unregistered central-only + /// The twelve replicated tables, plus the two deliberately-unregistered central-only /// notification tables, plus the node-local (also deliberately-unregistered) /// mirrored_secrets ledger. /// @@ -221,7 +221,7 @@ public class SiteLocalDbCdcRegistrationTests : IDisposable "OperationTracking", "site_events", "sf_messages", "deployed_configurations", "static_attribute_overrides", "shared_scripts", "external_systems", "database_connections", "data_connection_definitions", "native_alarm_state", - "deployed_secrets", + "deployed_secrets", "secret_mirror_status", "notification_lists", "smtp_configurations", "mirrored_secrets", ]; @@ -230,7 +230,7 @@ public class SiteLocalDbCdcRegistrationTests : IDisposable "OperationTracking", "site_events", "sf_messages", "deployed_configurations", "static_attribute_overrides", "shared_scripts", "external_systems", "database_connections", "data_connection_definitions", "native_alarm_state", - "deployed_secrets", + "deployed_secrets", "secret_mirror_status", ]; private static void AssertCaptureTriggersCoverTheReplicatedTables(ILocalDb db) diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbReplicatedTableFkGuardTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbReplicatedTableFkGuardTests.cs index 120e30b1..1ecc0784 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbReplicatedTableFkGuardTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbReplicatedTableFkGuardTests.cs @@ -15,8 +15,9 @@ namespace ZB.MOM.WW.ScadaBridge.Host.Tests; /// RegisterReplicated time, which on a site node is a BOOT failure of the replicating /// pair. This test moves that discovery from the first replicated boot to the build: it walks /// the real site schema (every table SiteLocalDbSetup.OnReady creates) and asserts none -/// of the eleven (incl. deployed_secrets, -/// added for site-secrets artifact deploy) has such an inbound FK. +/// of the twelve (incl. deployed_secrets, +/// added for site-secrets artifact deploy, and secret_mirror_status, added by review +/// finding M1) has such an inbound FK. /// /// /// The schema is built with an UNREPLICATED config on purpose: a replicated one would make diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs index 74e78ae4..f5a473cd 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs @@ -254,7 +254,7 @@ public class SiteLocalDbWiringTests : IDisposable "deployed_configurations", "static_attribute_overrides", "shared_scripts", "external_systems", "database_connections", "notification_lists", "data_connection_definitions", "smtp_configurations", "native_alarm_state", - "deployed_secrets", "mirrored_secrets", + "deployed_secrets", "mirrored_secrets", "secret_mirror_status", }) { Assert.Contains(table, tables); @@ -262,9 +262,10 @@ public class SiteLocalDbWiringTests : IDisposable } [Fact] - public void Site_LocalDb_RegistersExactlyTheElevenReplicatedTables() + public void Site_LocalDb_RegistersExactlyTheTwelveReplicatedTables() { - // The Phase 2 cutover, extended by site-secrets artifact deploy's deployed_secrets. + // The Phase 2 cutover, extended by site-secrets artifact deploy's deployed_secrets + // and review finding M1's per-node secret_mirror_status. // This is an EXACTLY check rather than a Contains in both directions, and both // directions are load-bearing. // @@ -285,8 +286,8 @@ public class SiteLocalDbWiringTests : IDisposable // data_connection_definitions precedes database_connections. "OperationTracking", "data_connection_definitions", "database_connections", "deployed_configurations", "deployed_secrets", "external_systems", - "native_alarm_state", "sf_messages", "shared_scripts", "site_events", - "static_attribute_overrides", + "native_alarm_state", "secret_mirror_status", "sf_messages", "shared_scripts", + "site_events", "static_attribute_overrides", ], db.ReplicatedTables.Keys.OrderBy(k => k, StringComparer.Ordinal).ToArray()); } @@ -334,6 +335,7 @@ public class SiteLocalDbWiringTests : IDisposable ["instance_unique_name", "source_canonical_name", "source_reference"], db.ReplicatedTables["native_alarm_state"].PkColumns); Assert.Equal(["name"], db.ReplicatedTables["deployed_secrets"].PkColumns); + Assert.Equal(["node_name"], db.ReplicatedTables["secret_mirror_status"].PkColumns); } private static HashSet TableNames(ILocalDb db) diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/DeployedSecretsStorageTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/DeployedSecretsStorageTests.cs index 79844668..29949c71 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/DeployedSecretsStorageTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/DeployedSecretsStorageTests.cs @@ -1,6 +1,7 @@ using System.Text.Json; using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts; +using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.TestSupport; @@ -303,6 +304,54 @@ public class DeployedSecretsStorageTests : IAsyncLifetime, IDisposable Assert.Equal("A", mirrored.Name); } + // ── secret_mirror_status (review finding M1) ── + + [Fact] + public async Task SecretMirrorStatus_RoundTrips_OneRowPerNode_OrderedByNodeName() + { + var at = new DateTimeOffset(2026, 9, 28, 6, 0, 0, TimeSpan.Zero); + await _storage.SetSecretMirrorStatusAsync( + "node-b", [new SecretMirrorFailure("A", "did not decrypt")], at); + await _storage.SetSecretMirrorStatusAsync("node-a", [], at.AddMinutes(1)); + + var rows = await _storage.GetSecretMirrorStatusesAsync(); + + Assert.Equal(["node-a", "node-b"], rows.Select(r => r.NodeName)); + Assert.Empty(rows[0].Failures); + Assert.Equal(at.AddMinutes(1), rows[0].CheckedAtUtc); + var failure = Assert.Single(rows[1].Failures); + Assert.Equal(new SecretMirrorFailure("A", "did not decrypt"), failure); + Assert.Equal(at, rows[1].CheckedAtUtc); + } + + [Fact] + public async Task SecretMirrorStatus_Set_UpsertsTheNodesRow() + { + var at = new DateTimeOffset(2026, 9, 28, 6, 0, 0, TimeSpan.Zero); + await _storage.SetSecretMirrorStatusAsync("node-a", [new SecretMirrorFailure("A", "x")], at); + await _storage.SetSecretMirrorStatusAsync("node-a", [], at.AddHours(1)); + + var row = Assert.Single(await _storage.GetSecretMirrorStatusesAsync()); + Assert.Empty(row.Failures); + Assert.Equal(at.AddHours(1), row.CheckedAtUtc); + } + + [Fact] + public async Task SecretMirrorStatus_UnparseableRow_ReadsAsAFailure_NotAsHealthy() + { + await using (var connection = _storage.CreateConnection()) + await using (var insert = connection.CreateCommand()) + { + insert.CommandText = + "INSERT INTO secret_mirror_status (node_name, failures_json, checked_at) VALUES ('node-b', '{not json', '2026-09-28T00:00:00.0000000+00:00')"; + await insert.ExecuteNonQueryAsync(); + } + + var row = Assert.Single(await _storage.GetSecretMirrorStatusesAsync()); + Assert.Equal("node-b", row.NodeName); + Assert.Single(row.Failures); + } + private static SecretArtifact Art(string name, long revision) => new( Name: name, ContentType: "application/octet-stream", diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/SiteStorageSchemaTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/SiteStorageSchemaTests.cs index 4816743c..99c23447 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/SiteStorageSchemaTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/SiteStorageSchemaTests.cs @@ -23,14 +23,14 @@ public class SiteStorageSchemaTests SiteStorageSchema.Apply(connection); SiteStorageSchema.Apply(connection); // second run must not throw - // Every table is named explicitly on purpose — a loop asserting "11 tables" + // Every table is named explicitly on purpose — a loop asserting "12 tables" // would still pass if one were renamed. foreach (var table in new[] { "deployed_configurations", "static_attribute_overrides", "shared_scripts", "external_systems", "database_connections", "notification_lists", "data_connection_definitions", "smtp_configurations", "native_alarm_state", - "deployed_secrets", "mirrored_secrets", + "deployed_secrets", "mirrored_secrets", "secret_mirror_status", }) { Assert.True(TableExists(connection, table), $"missing table: {table}"); diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Secrets/DeployedSecretsMirrorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Secrets/DeployedSecretsMirrorTests.cs index f7f728a5..1b694249 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Secrets/DeployedSecretsMirrorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Secrets/DeployedSecretsMirrorTests.cs @@ -2,6 +2,7 @@ using System.Collections.Concurrent; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging.Abstractions; using Microsoft.Extensions.Options; +using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services; using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Secrets; @@ -655,6 +656,132 @@ public class DeployedSecretsMirrorTests : IDisposable } } + // ── Per-node status row for central (review finding M1) ── + + [Fact] + public async Task Status_FirstPassAfterStartup_WritesThisNodesRow_EvenWhenHealthy() + { + var time = new FakeTimeProvider(new DateTimeOffset(2026, 9, 28, 6, 0, 0, TimeSpan.Zero)); + var (mirror, storage, _, _, _) = await CreateAsync(time: time, nodeIdentity: new FixedNodeIdentity("node-b")); + + await mirror.ReconcileAsync(CancellationToken.None); + + var status = Assert.Single(await storage.GetSecretMirrorStatusesAsync()); + Assert.Equal("node-b", status.NodeName); + Assert.Empty(status.Failures); + Assert.Equal(time.Now, status.CheckedAtUtc); + } + + [Fact] + public async Task Status_UnchangedFailureSet_IsNotRewrittenOnLaterPasses() + { + var time = new FakeTimeProvider(new DateTimeOffset(2026, 9, 28, 6, 0, 0, TimeSpan.Zero)); + var (mirror, storage, _, resolver, _) = await CreateAsync(time: time, nodeIdentity: new FixedNodeIdentity("node-b")); + resolver.ThrowFor("a"); + await storage.UpsertDeployedSecretsAsync([Art("a", 1, kek: "kek-central")], "d1"); + + await mirror.ReconcileAsync(CancellationToken.None); + var firstWrite = Assert.Single(await storage.GetSecretMirrorStatusesAsync()); + + // Same failure on every later pass: the row is replicated, so it must not be rewritten. + time.Now = time.Now.AddMinutes(5); + await mirror.ReconcileAsync(CancellationToken.None); + await mirror.ReconcileAsync(CancellationToken.None); + + var after = Assert.Single(await storage.GetSecretMirrorStatusesAsync()); + Assert.Equal(firstWrite.CheckedAtUtc, after.CheckedAtUtc); + var failure = Assert.Single(after.Failures); + Assert.Equal("a", failure.SecretName); + Assert.Contains("kek-central", failure.Message); + } + + [Fact] + public async Task Status_FailureSetChange_RewritesTheRow_AndClearsWhenHealed() + { + var time = new FakeTimeProvider(new DateTimeOffset(2026, 9, 28, 6, 0, 0, TimeSpan.Zero)); + var (mirror, storage, _, resolver, _) = await CreateAsync(time: time, nodeIdentity: new FixedNodeIdentity("node-a")); + await storage.UpsertDeployedSecretsAsync([Art("a", 1)], "d1"); + + await mirror.ReconcileAsync(CancellationToken.None); + Assert.Empty(Assert.Single(await storage.GetSecretMirrorStatusesAsync()).Failures); + + // A decrypt failure appears -> row rewritten with the failure. + resolver.ThrowFor("a"); + time.Now = time.Now.AddMinutes(1); + await mirror.ReconcileAsync(CancellationToken.None); + var failing = Assert.Single(await storage.GetSecretMirrorStatusesAsync()); + Assert.Equal("a", Assert.Single(failing.Failures).SecretName); + Assert.Equal(time.Now, failing.CheckedAtUtc); + + // Healed -> row rewritten empty. + resolver.ClearFailures(); + time.Now = time.Now.AddMinutes(1); + await mirror.ReconcileAsync(CancellationToken.None); + var healed = Assert.Single(await storage.GetSecretMirrorStatusesAsync()); + Assert.Empty(healed.Failures); + Assert.Equal(time.Now, healed.CheckedAtUtc); + } + + [Fact] + public async Task Status_RowCarriesNamesAndMessagesOnly_NeverCiphertext() + { + var (mirror, storage, _, resolver, _) = await CreateAsync(nodeIdentity: new FixedNodeIdentity("node-a")); + resolver.ThrowFor("a"); + await storage.UpsertDeployedSecretsAsync([Art("a", 1)], "d1"); + + await mirror.ReconcileAsync(CancellationToken.None); + + await using var connection = storage.CreateConnection(); + await using var command = connection.CreateCommand(); + command.CommandText = "SELECT failures_json FROM secret_mirror_status WHERE node_name = 'node-a'"; + var json = (string)(await command.ExecuteScalarAsync())!; + Assert.Contains("\"SecretName\":\"a\"", json); + Assert.DoesNotContain("Ciphertext", json); + Assert.DoesNotContain(Convert.ToBase64String(new byte[] { 1, 2, 3, 4, 5 }), json); + } + + [Fact] + public async Task Status_WriteFailure_DoesNotBreakThePass_AndIsRetriedNextPass() + { + var logger = new CapturingLogger(); + var (mirror, storage, store, _, _) = await CreateAsync(logger: logger, nodeIdentity: new FixedNodeIdentity("node-a")); + await storage.UpsertDeployedSecretsAsync([Art("a", 1)], "d1"); + + await using (var connection = storage.CreateConnection()) + await using (var drop = connection.CreateCommand()) + { + drop.CommandText = "DROP TABLE secret_mirror_status"; + await drop.ExecuteNonQueryAsync(); + } + + var r = await mirror.ReconcileAsync(CancellationToken.None); + + Assert.Equal(["a"], r.Applied); + Assert.True(store.Contains("a")); + Assert.Contains(logger.Entries, e => e.Level == LogLevel.Warning && e.Message.Contains("mirror status")); + + // Table back (e.g. transient fault cleared): the unchanged failure set is still + // written, because the earlier write never landed. + await storage.InitializeAsync(); + await mirror.ReconcileAsync(CancellationToken.None); + Assert.Equal("node-a", Assert.Single(await storage.GetSecretMirrorStatusesAsync()).NodeName); + } + + [Fact] + public async Task Status_WithoutNodeIdentity_FallsBackToMachineName() + { + var (mirror, storage, _, _, _) = await CreateAsync(); + + await mirror.ReconcileAsync(CancellationToken.None); + + Assert.Equal(Environment.MachineName, Assert.Single(await storage.GetSecretMirrorStatusesAsync()).NodeName); + } + + private sealed class FixedNodeIdentity(string? name) : INodeIdentityProvider + { + public string? NodeName { get; } = name; + } + // ── Harness ── private async Task<( @@ -665,7 +792,8 @@ public class DeployedSecretsMirrorTests : IDisposable FakeSecretCacheInvalidator Invalidator)> CreateAsync( CapturingLogger? logger = null, TimeSpan? interval = null, - TimeProvider? time = null) + TimeProvider? time = null, + INodeIdentityProvider? nodeIdentity = null) { var storage = new SiteStorageService(_localDb.Db, NullLogger.Instance); await storage.InitializeAsync(); @@ -685,7 +813,8 @@ public class DeployedSecretsMirrorTests : IDisposable invalidator, options, (ILogger?)logger ?? NullLogger.Instance, - time ?? TimeProvider.System); + time ?? TimeProvider.System, + nodeIdentity); return (mirror, storage, store, resolver, invalidator); } diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Secrets/SecretMirrorStatusReporterTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Secrets/SecretMirrorStatusReporterTests.cs new file mode 100644 index 00000000..42790b9c --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Secrets/SecretMirrorStatusReporterTests.cs @@ -0,0 +1,95 @@ +using Microsoft.Extensions.Logging.Abstractions; +using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health; +using ZB.MOM.WW.ScadaBridge.HealthMonitoring; +using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; +using ZB.MOM.WW.ScadaBridge.SiteRuntime.Secrets; +using ZB.MOM.WW.ScadaBridge.TestSupport; + +namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Secrets; + +/// +/// Review finding M1: forwards every node's +/// replicated secret_mirror_status row onto the site health report via the collector. +/// +public class SecretMirrorStatusReporterTests : IAsyncLifetime, IDisposable +{ + private readonly TestLocalDb _localDb = TestLocalDb.CreateTemp("secret-mirror-status"); + private SiteStorageService _storage = null!; + + public async Task InitializeAsync() + { + _storage = new SiteStorageService(_localDb.Db, NullLogger.Instance); + await _storage.InitializeAsync(); + } + + public Task DisposeAsync() => Task.CompletedTask; + + public void Dispose() + { + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); + } + + [Fact] + public async Task Probe_PushesEveryNodesRow_OntoTheNextReport() + { + var at = new DateTimeOffset(2026, 9, 28, 6, 0, 0, TimeSpan.Zero); + await _storage.SetSecretMirrorStatusAsync("node-a", [], at); + await _storage.SetSecretMirrorStatusAsync( + "node-b", [new SecretMirrorFailure("OPC-PWD", "did not decrypt on this node")], at); + var collector = new SiteHealthCollector(); + var reporter = new SecretMirrorStatusReporter( + collector, _storage, NullLogger.Instance); + + await reporter.ProbeAsync(); + var report = collector.CollectReport("site-a"); + + Assert.NotNull(report.SecretMirrorStatuses); + Assert.Equal(["node-a", "node-b"], report.SecretMirrorStatuses!.Select(s => s.NodeName)); + Assert.Equal("OPC-PWD", Assert.Single(report.SecretMirrorStatuses[1].Failures).SecretName); + } + + [Fact] + public void BeforeAnyProbe_TheReportFieldIsNull() + { + var report = new SiteHealthCollector().CollectReport("site-a"); + + Assert.Null(report.SecretMirrorStatuses); + } + + [Fact] + public async Task Probe_NoRowsYet_PushesAnEmptyList_NotNull() + { + var collector = new SiteHealthCollector(); + var reporter = new SecretMirrorStatusReporter( + collector, _storage, NullLogger.Instance); + + await reporter.ProbeAsync(); + + Assert.Empty(collector.CollectReport("site-a").SecretMirrorStatuses!); + } + + [Fact] + public async Task Probe_ReadFailure_KeepsThePreviousSnapshot_AndDoesNotThrow() + { + await _storage.SetSecretMirrorStatusAsync("node-a", [], DateTimeOffset.UtcNow); + var collector = new SiteHealthCollector(); + var reporter = new SecretMirrorStatusReporter( + collector, _storage, NullLogger.Instance); + await reporter.ProbeAsync(); + + await using (var connection = _storage.CreateConnection()) + await using (var drop = connection.CreateCommand()) + { + drop.CommandText = "DROP TABLE secret_mirror_status"; + await drop.ExecuteNonQueryAsync(); + } + + await reporter.ProbeAsync(); + + // Point-in-time: not drained on collect, and not cleared by a failed read. + Assert.Single(collector.CollectReport("site-a").SecretMirrorStatuses!); + Assert.Single(collector.CollectReport("site-a").SecretMirrorStatuses!); + } +} From f5756affefaa10331e125bfd603ba688a41da34f Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 28 Sep 2026 21:33:10 -0400 Subject: [PATCH 2/2] docs(site-secrets): document secret_mirror_status and standby mirror failures on the health report (M1) --- CLAUDE.md | 10 +++++----- .../2026-09-28-site-secrets-artifact-deploy-design.md | 9 +++++++++ docs/requirements/Component-CentralUI.md | 1 + docs/requirements/Component-HealthMonitoring.md | 2 ++ docs/requirements/Component-SiteRuntime.md | 4 +++- 5 files changed, 20 insertions(+), 6 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index b15e2431..9f239cc9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -90,7 +90,7 @@ spec for each is `docs/requirements/Component-.md`, and `README.md` carrie - DCL write failures returned synchronously to calling script. - Tag path resolution retried periodically for devices still booting. - Static attribute writes persisted to local SQLite (survive restart/failover, reset on redeployment). -- **Consolidated site database (LocalDb Phase 1 + 2, complete 2026-07-20).** Eleven tables now live in ONE `ZB.MOM.WW.LocalDb`-managed SQLite file (the `SiteLocalDbSetup.ReplicatedTables` list — the 11th, `deployed_secrets`, was added 2026-09-28 by the site-secrets-via-artifact-deploy change, see *Templates & Deployment* below), configured by the **required** `LocalDb:Path` (`/app/data/site-localdb.db` on the rig; validated with `ValidateOnStart`, so a site config missing it fails to boot). Consequences worth knowing: +- **Consolidated site database (LocalDb Phase 1 + 2, complete 2026-07-20).** Twelve tables now live in ONE `ZB.MOM.WW.LocalDb`-managed SQLite file (the `SiteLocalDbSetup.ReplicatedTables` list — the 11th, `deployed_secrets`, was added 2026-09-28 by the site-secrets-via-artifact-deploy change, see *Templates & Deployment* below; the 12th, `secret_mirror_status` — one row per node carrying that node's deployed-secrets mirror failures so the standby's reach central on the active node's health report — by review finding M1 the same day), configured by the **required** `LocalDb:Path` (`/app/data/site-localdb.db` on the rig; validated with `ValidateOnStart`, so a site config missing it fails to boot). Consequences worth knowing: - `site_events.id` changed from autoincrement INTEGER to an application-minted **GUID**. Last-writer-wins keys on the primary key, so two nodes independently minting `id=1,2,3…` would destroy each other's events rather than merge them. The event-log read path uses a composite `(timestamp, id)` keyset cursor with an **opaque string** continuation token; `EventLogEntry.Id` and both `ContinuationToken`s are `string`/`string?` on the site↔central Akka DTOs. - `ScadaBridge:OperationTracking:ConnectionString`, `ScadaBridge:SiteEventLog:DatabasePath` and — as of Phase 2 — `ScadaBridge:StoreAndForward:SqliteDbPath` + `ScadaBridge:Database:SiteDbPath` are all **migration-only** — nothing reads them but `SiteLocalDbLegacyMigrator`, which copies a pre-Phase-1 file in once (deterministic `mig-{NodeName}-{legacyId}` event ids, `INSERT OR IGNORE`, runs AFTER `RegisterReplicated` so migrated rows replicate) and renames it `.migrated`. Delete the keys once a node has migrated. - This incidentally fixes a data-loss bug: both legacy databases defaulted to CWD-relative paths **outside** the mounted volume and were discarded on every container recreate. @@ -101,10 +101,10 @@ spec for each is `docs/requirements/Component-.md`, and `README.md` carrie - **`notification_lists` and `smtp_configurations` are created but deliberately NOT registered.** They are permanently empty on a site (no writer since 2026-07-10, the migrator skips them, the active-node purge keeps them empty), and registering them would open a standing replication channel whose only historical payload was plaintext SMTP passwords. Pinned by a security-named test, and verified live: those two tables have **no CDC triggers** on either rig node. - **Operational constraints (read before upgrading a site pair):** stop and start both nodes TOGETHER — rolling one at a time is no longer supported, since the legacy `SfBufferSnapshot` compatibility handler went with the replicator. And a node offline longer than `LocalDb:Replication:TombstoneRetention` (default 7 days) can resurrect deleted rows on rejoin. See `docs/deployment/topology-guide.md`. - **Batching is by BYTE BUDGET as of LocalDb 0.2.0** — `LocalDb:Replication:MaxBatchBytes` (default **2 MB**, sized under the 4 MB gRPC cap) bounds a delta/snapshot message by summed serialized size via a per-message split in `SyncSession.PumpLoopAsync`, with `MaxBatchSize` demoted to a secondary row cap; a single row over budget is sent alone rather than stalling the stream. **The rig's old `MaxBatchSize = 16` pin (a hand-computed byte-budget proxy: ~70 KB worst-case `config_json` x the 500 default is ~35 MB) is retired, but `MaxBatchSize` is NOT fully redundant with `MaxBatchBytes`** — it also bounds the separate DB READ page in `OplogStore.ReadBatchAboveAsync`/`SnapshotStreamer`, which materializes the whole page into memory *before* the byte-budget split runs, so an unset (500-default) `MaxBatchSize` still lets a reconnect drain transiently allocate ~35 MB per read even though every wire message stays under `MaxBatchBytes` (arch-review adversarial finding F2). Both site-a nodes on `docker/` therefore pin an explicit `"MaxBatchSize": 64` to bound that transient allocation, while `MaxBatchBytes` stays unset (its 2 MB default) to bound the wire message; site-b/site-c stay unreplicated so the key doesn't apply there. - - **CDC registration is conditional, and both directions self-heal at boot** (arch-review WP1.3 + WP3.3, `Host/SiteLocalDbSetup.cs`). Capture triggers are installed only when this node has replication configured — `PeerAddress` **or** `ApiKey`, an OR because only the dialling half sets `PeerAddress` while the passive half carries the key alone. An unreplicated node calls **`DeregisterReplicated`** on all eleven tables at boot, dropping triggers an earlier build left behind and pruning their oplog/row-version rows (idempotent; logs once at Information when something was actually cleaned). A replicating node registers with **`baselineExistingRows: true`**, which seeds `__localdb_row_version` for pre-existing rows at the LWW floor (HLC `0`, this node's id) and flags a snapshot resync — so turning replication ON for a site that has been running without it now converges on the rows already in the file instead of only on writes made after the restart. Deregistration must be symmetric (the handshake compares registered-table digests fail-closed), so replication is a both-nodes-together change in either direction. LocalDb 0.2.0 also makes backlog depth O(1) and drops the unused `__localdb_oplog_hlc` index; the on-disk bookkeeping schema goes to v2, upgraded in place on open, with no wire change (0.1.x peers still sync). + - **CDC registration is conditional, and both directions self-heal at boot** (arch-review WP1.3 + WP3.3, `Host/SiteLocalDbSetup.cs`). Capture triggers are installed only when this node has replication configured — `PeerAddress` **or** `ApiKey`, an OR because only the dialling half sets `PeerAddress` while the passive half carries the key alone. An unreplicated node calls **`DeregisterReplicated`** on all twelve tables at boot, dropping triggers an earlier build left behind and pruning their oplog/row-version rows (idempotent; logs once at Information when something was actually cleaned). A replicating node registers with **`baselineExistingRows: true`**, which seeds `__localdb_row_version` for pre-existing rows at the LWW floor (HLC `0`, this node's id) and flags a snapshot resync — so turning replication ON for a site that has been running without it now converges on the rows already in the file instead of only on writes made after the restart. Deregistration must be symmetric (the handshake compares registered-table digests fail-closed), so replication is a both-nodes-together change in either direction. LocalDb 0.2.0 also makes backlog depth O(1) and drops the unused `__localdb_oplog_hlc` index; the on-disk bookkeeping schema goes to v2, upgraded in place on open, with no wire change (0.1.x peers still sync). - **LocalDb 0.3.0 (2026-08-15) makes that snapshot flag PER-TABLE** — the residual 0.2.1 explicitly deferred. On-disk bookkeeping schema goes to **v3** (a new `__localdb_snapshot_state` table, upgraded in place on open); still wire-compatible, and `lib_schema_version` deliberately stays `1` because the handshake compares it fail-closed for *equality*. The extension is negotiated by **capability** (`Handshake.supports_partial_snapshot`, `SnapshotBegin.tables` where **empty == all tables**), so a peer that does not advertise it is served a full snapshot — a scoped one would make it advance `last_applied_remote_seq` past deltas it never received. For the same reason **a partial snapshot moves no watermark on either side**; the overlapping deltas are still pumped and absorbed by LWW. Snapshot debt now clears on the receiver's new `SnapshotAck` rather than on enqueue, so a crash mid-snapshot re-streams the same scope. **`SiteLocalDbSetup.cs` needed no restructuring** — it already calls `RegisterReplicated` once per table in a loop, which is exactly what per-table flagging keys off. The consequence to know: **adding an eleventh table to `ReplicatedTables` on an already-replicating site now snapshots that one table**, where through 0.2.x the same edit re-streamed all eleven in full in both directions. First-boot behaviour is unchanged (all ten seed at once ⇒ the flagged set is every registered table ⇒ an ordinary full snapshot), and so is upgrading the rig in place (the ten tables are already ledgered, nothing seeds, nothing is flagged). A future "register the whole list in one call" API would silently give the coarse behaviour back. - **LocalDb 0.3.2 (2026-08-19, ScadaBridge arch-review P4-1 + P4-3; wire-compatible, no schema change):** (1) the sync pump now **faults the live session when a cap-prune invalidates its position** (horizon check on the oplog's min seq before each read batch + on the batch head after it; throws the new public `LocalDbOplogGapException` through the 0.3.1 fault-teardown) — previously a slow-but-alive peer silently skipped the pruned gap and the pair diverged until the next reconnect; (2) **transport keepalives both sides** — client `SocketsHttpHandler` pings via new validated `LocalDb:Replication:KeepAlivePingDelay`/`KeepAlivePingTimeout` (defaults 60s/20s, `Always` policy), and ScadaBridge's site gRPC listener sets Kestrel h2 keepalive (`Host/SiteKestrelSetup.cs` — server-wide, also governs the metrics port); (3) the server's single-stream guard is now **newest-wins with an awaited handover** — a fresh authenticated `Sync` cancels a dead/wedged incumbent and *waits* for it to finish (`TakeoverTimeout` 10s) before starting, so a half-open link can no longer wedge the pair behind `AlreadyExists` while the zombie side reports `LocalDbReplicationConnected=true`. - - **LocalDb 0.3.3 (2026-08-20, ScadaBridge arch-review P4-7; wire-compatible, no schema change — superseded by 0.3.4 below in the same bullet):** two latent `LwwApplier` hazards closed before any table could hit them. (1) **`RegisterReplicated` now REFUSES a table that is the parent of a foreign key with a row-mutating `ON DELETE` action (`CASCADE`/`SET NULL`/`SET DEFAULT`) from any other table** (`LocalDbRegistrationException` naming `child.column ON DELETE `; `RESTRICT`/`NO ACTION` still allowed) — the applier's `INSERT OR REPLACE` runs with `foreign_keys=ON`, so the implicit delete fired those actions while CDC capture was suppressed, i.e. child rows changed on one node and never replicated. Registration-time only: SQLite can't add an FK to an existing table, so a later-created child is caught at the next boot's registration. None of ScadaBridge's eleven tables has an FK (verified on the live rig file: zero `REFERENCES` across 29 `CREATE TABLE`s) — **keep it that way; a cascading-FK parent can never be one of `ReplicatedTables`** (register row 105). (2) The per-entry apply (delete-or-upsert + row-version ledger write) is now **atomic under a `SAVEPOINT`** — a fault between the two statements used to leave a deleted row with a live ledger entry, which `SnapshotStreamer` warns-and-skips forever; poison-catch semantics unchanged (narrow `SqliteException`, per-entry dead-letter, batch continues). Cost: two extra in-memory statements per applied entry on a backlog drain. **Both are pinned here** (2026-08-20): `SiteLocalDbReplicatedTableFkGuardTests` asserts at build time that no `ReplicatedTables` entry has an inbound row-mutating FK (register row 105), and **pre-2026-08-20 site files must be VACUUMed once offline** to get `auto_vacuum=INCREMENTAL` — `docker/vacuum-site-localdb.sh` (pair stopped together; done on all three rig pairs; topology guide → *one-off offline VACUUM*; row 104). **LocalDb 0.3.4 (2026-08-20, register row 106; wire-compatible, sender-side only):** a converged pair whose sender oplog was ack-pruned to empty used to stamp snapshots `as_of_seq 0`, so a peer with a zero watermark (a rebuilt node — the rig's site-a-a since 2026-08-15) received a FULL snapshot on every reconnect until the sender wrote again (127 k rows per restart, observed). The as-of now floors at `last_acked_seq`; proven live — second restart after the pin exchanged zero snapshots. Lesson for rebuilding a node from its peer: the first reconnect snapshots, the second must not. + - **LocalDb 0.3.3 (2026-08-20, ScadaBridge arch-review P4-7; wire-compatible, no schema change — superseded by 0.3.4 below in the same bullet):** two latent `LwwApplier` hazards closed before any table could hit them. (1) **`RegisterReplicated` now REFUSES a table that is the parent of a foreign key with a row-mutating `ON DELETE` action (`CASCADE`/`SET NULL`/`SET DEFAULT`) from any other table** (`LocalDbRegistrationException` naming `child.column ON DELETE `; `RESTRICT`/`NO ACTION` still allowed) — the applier's `INSERT OR REPLACE` runs with `foreign_keys=ON`, so the implicit delete fired those actions while CDC capture was suppressed, i.e. child rows changed on one node and never replicated. Registration-time only: SQLite can't add an FK to an existing table, so a later-created child is caught at the next boot's registration. None of ScadaBridge's twelve replicated tables has an FK (verified on the live rig file: zero `REFERENCES` across 29 `CREATE TABLE`s) — **keep it that way; a cascading-FK parent can never be one of `ReplicatedTables`** (register row 105). (2) The per-entry apply (delete-or-upsert + row-version ledger write) is now **atomic under a `SAVEPOINT`** — a fault between the two statements used to leave a deleted row with a live ledger entry, which `SnapshotStreamer` warns-and-skips forever; poison-catch semantics unchanged (narrow `SqliteException`, per-entry dead-letter, batch continues). Cost: two extra in-memory statements per applied entry on a backlog drain. **Both are pinned here** (2026-08-20): `SiteLocalDbReplicatedTableFkGuardTests` asserts at build time that no `ReplicatedTables` entry has an inbound row-mutating FK (register row 105), and **pre-2026-08-20 site files must be VACUUMed once offline** to get `auto_vacuum=INCREMENTAL` — `docker/vacuum-site-localdb.sh` (pair stopped together; done on all three rig pairs; topology guide → *one-off offline VACUUM*; row 104). **LocalDb 0.3.4 (2026-08-20, register row 106; wire-compatible, sender-side only):** a converged pair whose sender oplog was ack-pruned to empty used to stamp snapshots `as_of_seq 0`, so a peer with a zero watermark (a rebuilt node — the rig's site-a-a since 2026-08-15) received a FULL snapshot on every reconnect until the sender wrote again (127 k rows per restart, observed). The as-of now floors at `last_acked_seq`; proven live — second restart after the pin exchanged zero snapshots. Lesson for rebuilding a node from its peer: the first reconnect snapshots, the second must not. - **LocalDb 0.2.1** (two adversarial-review findings against 0.2.0, wire-compatible, no schema change) fixes: (1) `DeregisterReplicated` was deleting the HLC clock's only durable crash anchor along with the table rows — an ungraceful exit (crash, SIGKILL, power loss) on a fully-deregistered file left nothing to recover from, so the reopened clock could restart from wall-clock time and re-issue stamps below HLCs peers already hold (silently discarded by LWW); the current clock is now flushed inside the deregistration transaction, before the prune. (2) The sync inbox was an **unbounded** channel, so inbound memory was a function of what the peer sends — a mutual full snapshot (both sides of a pair baseline unconditionally when replication is enabled late) buffered whole, in memory, on both nodes at once; it is now **bounded at 64 messages** with wait-mode backpressure (unrelated to, and not to be confused with, the rig's separate `MaxBatchSize: 64` DB-read-page pin above), which required moving the snapshot to run alongside the receive loop (not inside the pre-loop handshake) so the inbox is drained for the whole session and can't deadlock itself. - All timestamps are UTC throughout the system. - Inter-cluster communication uses **three** transports, not two — **all cross-cluster command/control and data now rides gRPC** after the ClusterClient→gRPC migration's Phase 4 (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`) deleted Akka `ClusterClient`/`ClusterClientReceptionist`: (1) **gRPC command/control** — site→central over the central-hosted `CentralControlService` (`GrpcCentralTransport`, sticky central-a→central-b channel pair; deployments/notifications/health/heartbeat/audit-ingest/reconcile), and central→site over the site-hosted `SiteCommandService` (`GrpcSiteTransport`, per-site NodeA→NodeB channel pair; the 28 lifecycle/OPC-UA/query/parked/route/failover commands); (2) **gRPC** server-streaming for real-time data (attribute values, alarm states, `SiteStreamService`); and (3) **plain token-gated HTTP** for the deployment config itself — notify-and-fetch, the site pulls the config from `DeploymentConfigEndpoints` (`ManagementService/DeploymentConfigEndpoints.cs`) with an `X-Deployment-Token` header, `AllowAnonymous` with the per-deployment token as the entire security boundary. The gRPC boundary is per-site PSK-authenticated (`ControlPlaneAuthInterceptor`, unchanged). There is **no receptionist registration** — discovery is by dialling configured endpoints; central builds one `SitePairChannelProvider` per site (addresses from `Site.GrpcNodeAAddress`/`GrpcNodeBAddress`, refreshed from the DB every 60s and on admin changes), sites dial `ScadaBridge:Communication:CentralGrpcEndpoints` (both central nodes, h2c on `CentralGrpcPort` 8083, **NOT** via Traefik). **Discovery is asymmetric by design:** central discovers site gRPC addresses from the *database* (refreshable at runtime), sites discover central from *appsettings* (`CentralGrpcEndpoints`, static — restart required; `StartupValidator` requires a Site node to list at least one). `Akka.Cluster.Tools` stays for ClusterSingleton; only the ClusterClient part is gone. **Central never buffers for an unreachable site** — the send fails with the caller's Ask/deadline timing out; a `ConnectionStateChanged` mechanism built for this was deleted as dead code. @@ -141,7 +141,7 @@ spec for each is `docs/requirements/Component-.md`, and `README.md` carrie - Per-instance operation lock covers all mutating commands (deploy, disable, enable, delete). - **Bulk site deployment (`DeploymentService.DeploySiteAsync`, arch-review WP2.5)** redeploys every out-of-date instance at a site through one flatten session (a template chain shared by N instances is walked once) in three phases — serial Prepare, bounded-parallel Send (`SiteDeploymentMaxParallelism`, default 4), serial Finalize — still **not all-or-nothing**: any one instance's failure doesn't block the rest. **Disabled instances are SKIPPED by bulk deploy as of 2026-08-19 (arch-review P7-2)** — they are reported in the summary's additive `SkippedInstances` list, never flattened/sent/re-enabled (single-instance deploy remains the explicit operator path that deploys+enables a Disabled instance); **Up-to-date instances are ALSO skipped as of 2026-08-20 (arch-review P7-6)** — a redeploy is not idempotent at the site (it resets static overrides and clears native alarm state), so `DeploySiteAsync` pre-filters through `IStaleInstanceProbe` (trustworthy since P7-4 fixed its memo timing) and reports unchanged instances in `SkippedInstances` with reason `up-to-date`; the filter is conservative (every uncertainty deploys: missing snapshot, `ShouldQuerySiteBeforeRedeploy`, null/faulting probe). Additive `MgmtDeploySiteCommand.Force` / CLI `deploy site --force` restores the unconditional push for what central cannot see (restored volume, rebuilt node) — **Disabled still wins under `--force`**. Also P7-5: the three phases share one scoped DbContext and a failed `SaveChanges` leaves the entity tracked, so one permanent failure used to cascade to every later instance and strand records `InProgress`; `MarkDeploymentFailedAsync` now detaches pending changes and retries the `Failed` write once on the clean tracker; Prepare's per-instance reconcile also runs under a short `SiteDeploymentReconcileTimeout` (5s) with a once-per-batch unreachable-site circuit-breaker, so a down site no longer costs 30s × N serial before the first send (P7-3). The `PendingDeployment` staging row (5-minute TTL) is written per-instance immediately before that instance's round-trip in the Send phase, not upfront in Prepare, so a large batch's tail instances don't have their TTL clock already half-spent by the time they're reached. - **Site artifact revision + fleet-wide pending-updates indicator (2026-09-14, `docs/plans/2026-09-14-pending-updates-design.md`).** `Site.ArtifactRevisionHash`/`ArtifactsDeployedAt` (migration `AddSiteArtifactRevision`) are stamped by `ArtifactDeploymentService` only on a site whose artifact deploy succeeds, so a redeployed fleet's artifacts are trackable per site the same way instance staleness already is. `PendingUpdatesEvaluator` combines that with `DeploymentService.GetInstanceStalenessAsync` (one shared `FlattenSession` for the whole fleet) into a `PendingUpdatesSnapshot`, held behind a process-wide singleton `IPendingUpdatesCache` (`ScadaBridge:DeploymentManager:PendingUpdatesCacheTtl`, default 60s) so N Blazor circuits cost one fleet flatten per TTL, not one each. Surfaces: the Sites page's per-site **Artifacts** badge (`Current`/`Out of date`/`Never deployed`) + instance-staleness line + header **Re-check**, and a rail-footer **pending updates** indicator (Deployer/Administrator only, 30s poll, site-scoped). **Upgrade consequence:** every site reads *Never deployed* until one artifact deploy runs post-upgrade — deliberate, not a bug. Also shipped: the rail nav now sits in its own scroll pane (`NavMenu`'s `.rail-scroll` + `site.css`) so a long nav can't push the footer/indicator off-screen; the `ZB.MOM.WW.Theme` kit itself is untouched (register row 123 proposes upstreaming it). -- **Site secrets ride the artifact deploy; the gRPC secrets hub is retired in ScadaBridge (2026-09-28, `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`).** `Deploy Artifacts` now ships each site **only the secrets it references** — `SiteSecretsCollector` scans a site's data-connection JSON for `secret:` (`SecretReferenceScanner`, protocol-agnostic, bootstrap names like `SB-GRPC-PSK-*` excluded) and resolves each against the central `ISecretStore`; a missing/tombstoned secret fails **that site's** deploy before anything is sent (fail-closed), ciphertext only ever crosses the wire, and secrets factor into `ArtifactRevisionHashService`'s staleness hash (a missing secret hashes as a distinguishable marker, so an unresolvable reference never reads *Current*; a site with no secret references hashes identically whether the field is null or empty). A pre-feature site (answers with no `SecretsApplied`) or a partial apply is also treated as a deploy failure and is never stamped; one `DeploySiteSecrets`/`SiteSecrets` audit row (`secrets` name + revision and `missing` names only, never ciphertext; no `removed` field — a removal-only deploy writes no central row) is written per site deploy that carried secrets or a missing-secret failure, on both the sweep and the per-site retry paths. Site-side: `deployed_secrets` is the 11th LocalDb-replicated table (ciphertext rows only, upsert-then-soft-remove); the per-node `DeployedSecretsMirror` (`SiteRuntime:DeployedSecretsMirrorInterval`, default 15s, also run synchronously right after an active-node apply) reconciles it into that node's local `ISecretStore`, ledgers what it mirrored in the node-local `mirrored_secrets` table so an out-of-band local secret sharing a deployed name is never silently touched (a mirror-owned name deleted or edited locally is healed back to the deployed value on the next pass, even at an unchanged revision), and trial-decrypts every pass to catch a KEK mismatch. Secrets are mirrored before the deploy's data connections reach the DCL; a rotation with unchanged connection config takes effect on the next (re)connect. **A site pair using deployed secrets must run LocalDb replication** — otherwise only the node active at deploy time holds them. Central runs one shared secret store, `Secrets:Store:Provider=SqlServer` (Central role only; a clustered central left on `Sqlite` gets a boot Warning) — the pre-host `${secret:}` expander reads the same store the running node serves. **ALL secrets replication is retired in ScadaBridge** — the pull-only gRPC hub and the SQL-Server replication mode both — `Secrets:Replication:Enabled` set to anything but empty/`false` now fails `StartupValidator` on either role with a message pointing at Deploy Artifacts / `Secrets:Store:Provider=SqlServer`. +- **Site secrets ride the artifact deploy; the gRPC secrets hub is retired in ScadaBridge (2026-09-28, `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`).** `Deploy Artifacts` now ships each site **only the secrets it references** — `SiteSecretsCollector` scans a site's data-connection JSON for `secret:` (`SecretReferenceScanner`, protocol-agnostic, bootstrap names like `SB-GRPC-PSK-*` excluded) and resolves each against the central `ISecretStore`; a missing/tombstoned secret fails **that site's** deploy before anything is sent (fail-closed), ciphertext only ever crosses the wire, and secrets factor into `ArtifactRevisionHashService`'s staleness hash (a missing secret hashes as a distinguishable marker, so an unresolvable reference never reads *Current*; a site with no secret references hashes identically whether the field is null or empty). A pre-feature site (answers with no `SecretsApplied`) or a partial apply is also treated as a deploy failure and is never stamped; one `DeploySiteSecrets`/`SiteSecrets` audit row (`secrets` name + revision and `missing` names only, never ciphertext; no `removed` field — a removal-only deploy writes no central row) is written per site deploy that carried secrets or a missing-secret failure, on both the sweep and the per-site retry paths. Site-side: `deployed_secrets` is the 11th LocalDb-replicated table (ciphertext rows only, upsert-then-soft-remove); the per-node `DeployedSecretsMirror` (`SiteRuntime:DeployedSecretsMirrorInterval`, default 15s, also run synchronously right after an active-node apply) reconciles it into that node's local `ISecretStore`, ledgers what it mirrored in the node-local `mirrored_secrets` table so an out-of-band local secret sharing a deployed name is never silently touched (a mirror-owned name deleted or edited locally is healed back to the deployed value on the next pass, even at an unchanged revision), and trial-decrypts every pass to catch a KEK mismatch. Each node also records its current mirror failures (names + messages only) in the 12th replicated table `secret_mirror_status` — written only when its failure set changes, plus once per start-up — and the active node forwards every node's row as `SiteHealthReport.SecretMirrorStatuses` (proto field 33), so a **standby** KEK mismatch shows as a *Secret mirror failing* badge on the Health dashboard instead of only in the standby's log (review finding M1). Secrets are mirrored before the deploy's data connections reach the DCL; a rotation with unchanged connection config takes effect on the next (re)connect. **A site pair using deployed secrets must run LocalDb replication** — otherwise only the node active at deploy time holds them. Central runs one shared secret store, `Secrets:Store:Provider=SqlServer` (Central role only; a clustered central left on `Sqlite` gets a boot Warning) — the pre-host `${secret:}` expander reads the same store the running node serves. **ALL secrets replication is retired in ScadaBridge** — the pull-only gRPC hub and the SQL-Server replication mode both — `Secrets:Replication:Enabled` set to anything but empty/`false` now fails `StartupValidator` on either role with a message pointing at Deploy Artifacts / `Secrets:Store:Provider=SqlServer`. - **Create Instance picks its template from the Templates folder tree** (2026-08-21): the flat `