feat(hosts): surface per-host connectivity on /hosts; drop the unwritable table (#521)

`IHostConnectivityProbe` was a dead surface: eleven drivers implement it, `GetHostStatuses()`
had ZERO production call sites, and `OnHostStatusChanged` had no subscriber outside the Galaxy
driver's own aggregator. Per-host connectivity was computed by every driver and read by nobody.

The issue offered "build the publisher or delete it". The publisher as its entity doc described
it — driver nodes upserting `DriverHostStatus` rows — is not buildable: per-cluster mesh Phase 4
gates `AddOtOpcUaConfigDb` on the `admin` role, so a driver-only node has no ConfigDb connection
to write rows with. So the capability is kept and the transport changed.

`DriverHealthChanged.HostStatuses` now carries the probe result to `/hosts` as a Hosts column.
That channel already reached the page, already survives the mesh split via the Phase 5 gRPC
telemetry stream, and already replays a last-value snapshot on re-subscribe — so per-host state
re-primes after a reconnect without a durable store. Both halves of the interface finally do
what they are for: the event triggers a prompt publish, the pull is the source of truth.

The point of the column is the case the driver-level state chip structurally cannot express: a
multi-device driver stays aggregate-Healthy while ONE of its devices is unreachable.

Two traps, both pinned by tests that were falsified against the prod code:

- The host digest MUST be in the publish fingerprint. On a single-host-down transition every
  other fingerprint component is unchanged, so the dedup would swallow exactly the publish
  carrying the news — the trap that already bit the rediscovery signal. Removing it turns the
  guard test red, verified.
- null (no probe) must stay distinct from empty (probe with no hosts). proto3 cannot tell an
  absent repeated field from an empty one, hence the explicit `has_host_statuses` flag;
  collapsing them would render every probe-less driver as one whose devices are all fine.

Dropped: the DriverHostStatus entity, enum, DbSet, model config and table (migration
DropDriverHostStatusTable — empty on every deployment, so the scaffolder's data-loss warning is
moot, and Down() recreates it exactly).

Found en route, NOT fixed here: `DriverInstanceResilienceStatus` is the identical defect — no
writer, no reader, only a DbSet declaration, while the live data rides the
`driver-resilience-status` telemetry channel. Its doc-comment now states that rather than
describing the sampler and AdminUI join that were never built. Filed as #524 rather than widening
this schema change beyond what was asked.

Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
This commit is contained in:
Joseph Doherty
2026-07-30 04:21:08 -04:00
parent dc9d947bca
commit 30d0697c28
24 changed files with 2411 additions and 305 deletions
@@ -3,6 +3,9 @@ using Shouldly;
using ZB.MOM.WW.OtOpcUa.Commons.Protos;
using ZB.MOM.WW.OtOpcUa.Commons.Protos.Telemetry.V1;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Telemetry;
// Aliased, not imported wholesale: Core.Abstractions also declares a DriverHealth, which collides with
// the proto DriverHealth these tests construct.
using HostState = ZB.MOM.WW.OtOpcUa.Core.Abstractions.HostState;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.Telemetry;
@@ -168,6 +171,77 @@ public sealed class TelemetryProtoMapCentralTests
e.PublishedUtc.Kind.ShouldBe(DateTimeKind.Utc);
}
/// <summary>
/// Per-host connectivity (Gitea #521) survives the wire with its tri-state intact:
/// <b>null</b> (driver has no probe) must stay distinguishable from an <b>empty list</b> (it has one
/// that knows no hosts). proto3 cannot tell an absent repeated field from an empty one, which is why
/// <c>has_host_statuses</c> exists — without it a driver with no probe would arrive looking like one
/// whose devices are all fine, and the /hosts column would render "0 hosts" for every driver in the
/// fleet.
/// </summary>
[Fact]
public void ToHealth_host_statuses_round_trip_with_the_null_vs_empty_distinction_intact()
{
var populated = new DriverHealth
{
ClusterId = "c1", DriverInstanceId = "d1", State = "Healthy",
PublishedUtc = Timestamp.FromDateTime(SampleUtc),
HasHostStatuses = true,
HostStatuses =
{
new HostConnectivity { HostName = "plc-a", State = "Running", LastChangedUtc = Timestamp.FromDateTime(OtherUtc) },
new HostConnectivity { HostName = "plc-b", State = "Stopped", LastChangedUtc = Timestamp.FromDateTime(SampleUtc) },
},
};
var mapped = TelemetryProtoMapCentral.ToHealth(populated).HostStatuses;
mapped.ShouldNotBeNull();
mapped!.Count.ShouldBe(2);
mapped[0].HostName.ShouldBe("plc-a");
mapped[0].State.ShouldBe(HostState.Running);
mapped[0].LastChangedUtc.ShouldBe(OtherUtc);
mapped[1].State.ShouldBe(HostState.Stopped);
// A probe that currently knows no hosts: empty, NOT null.
var empty = new DriverHealth
{
ClusterId = "c1", DriverInstanceId = "d1", State = "Healthy",
PublishedUtc = Timestamp.FromDateTime(SampleUtc),
HasHostStatuses = true,
};
TelemetryProtoMapCentral.ToHealth(empty).HostStatuses.ShouldBeEmpty();
// No probe at all: null, NOT empty.
var absent = new DriverHealth
{
ClusterId = "c1", DriverInstanceId = "d1", State = "Healthy",
PublishedUtc = Timestamp.FromDateTime(SampleUtc),
};
TelemetryProtoMapCentral.ToHealth(absent).HostStatuses.ShouldBeNull();
}
/// <summary>
/// An unparseable host state degrades to <see cref="HostState.Unknown"/> rather than throwing. A node
/// running a newer build that added an enum member must not be able to kill central's telemetry
/// stream — this is observability, and it has no business failing closed.
/// </summary>
[Fact]
public void ToHealth_unknown_host_state_string_degrades_instead_of_throwing()
{
var proto = new DriverHealth
{
ClusterId = "c1", DriverInstanceId = "d1", State = "Healthy",
PublishedUtc = Timestamp.FromDateTime(SampleUtc),
HasHostStatuses = true,
HostStatuses = { new HostConnectivity { HostName = "plc-a", State = "Quiescing" } },
};
var mapped = TelemetryProtoMapCentral.ToHealth(proto).HostStatuses;
mapped.ShouldNotBeNull();
mapped![0].State.ShouldBe(HostState.Unknown);
}
[Fact]
public void ToHealth_absent_nullable_timestamp_and_optional_string_map_to_null()
{