Every deploy stored the whole gzip'd fleet snapshot in Deployment.ArtifactBlob forever (R03:1.2).
DeploymentRetentionActor (admin singleton, proxy-less) sweeps every Deployments:Retention:
IntervalMinutes (60): terminal rows (Sealed / PartiallyFailed / TimedOut) older than the newest
KeepSealed (50) and not referenced by a NodeDeploymentState started/applied within ReferencedDays
(30) get their blob EMPTIED — never deleted, never NULLed: the column stays NOT NULL and every
reader already treats zero bytes as "no artifact" (#485), so no schema migration and no reader
change. In-flight rows are never touched; MaxRowsPerSweep (200) bounds one sweep.
Refs #733
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
Clears CommentChecker findings in src/ (3,280 -> 209, the remainder being false
positives like PLC-5/IEEE-754/M0.0) by adding missing summaries/params/returns,
normalising inheritdoc usage, and removing Gitea/review tracking IDs while keeping
the rationale they annotated. Comment-only: comment-stripped sources are identical
to the parent commit.
MeshGrpcBoundaryTests replaces the MeshClusterClientBoundaryTests deleted in step 1. Three
self-seeded single-member clusters sharing the ActorSystem name, joined ONLY by gRPC, driving
the REAL CentralCommunicationActor -> MeshGrpcCommandDialer -> Kestrel h2c ->
MeshCommandAuthInterceptor -> NodeCommandGrpcService -> NodeCommunicationActor -> EventStream:
1. No_cross_mesh_Akka_association_forms_between_central_and_either_site — the program's own
point, kept as a test rather than a live-gate note: each cluster stays single-member and
none ever holds another's address as a member.
2. One_dispatch_through_central_reaches_a_node_in_every_cluster_mesh — the fan, with each
site holding its OWN per-cluster PSK, so a dial that selected the wrong cluster's key
would be refused at the far side.
3. Falsifiability_control_a_dial_carrying_the_wrong_cluster_key_delivers_nothing — without
it, (2) cannot distinguish "the gRPC boundary carried it" from "something else in the
process did". The same node then accepts the right key, so the silence is the auth gate
and not a dead endpoint.
All three run fully offline — no Docker fixture — and pass.
Comment sweep across every remaining clusterclient|receptionist hit in src/ and tests/:
the program label becomes "mesh gRPC command plane", and every comment that described the
deleted transport as live now describes what actually runs (Dps = pair-local default,
Grpc = cross-mesh). mesh_transport.proto's header/service comments are reworded; no message,
field or enum is touched — the contract is locked. MeshCommActorPathTests is reshaped rather
than retired: the two paths still exist and still matter, but what a mismatch costs is now a
misleading Unavailable from the gRPC ingress rather than a silently dropped receptionist send.
The only surviving mentions of the old names are the ones that must survive: the fail-closed
rules that name "ClusterClient" as a retired value refused at boot, their tests, and this
test class's own account of what it replaces.
Stage 1b. Still no enforcement wired — this adds the credential model the
identity cutover (stage 2) will read.
ApiKey holds one principal per key, deliberately. The reason to separate keys
from principals is so many keys share a grant set and rotating a key never
touches an ACL row, and that holds with one principal per key since nothing stops
two keys naming the same principal. A key belonging to several principals at once
buys little and costs a join table. UserAuthorizationState.Principals stays a
list, so adding it later needs no ACL-side change.
ApiKeySecret is the security-critical half and is pure, so it is tested on its
own: otua_<keyId>_<secret>, peppered HMAC-SHA256, FixedTimeEquals. Notes why a
plain HMAC rather than an iterated KDF (the secret is 32 bytes of CSPRNG, and the
call runs inline on session activation, which the SDK serializes server-wide
behind its SessionManager event lock), and fails closed on an unset pepper rather
than hashing under an empty key.
Found and fixed a bug in my own first draft: the key id was url-safe base64,
whose alphabet maps '/' to '_' — the very character TryParse splits on. Roughly
one key in five would have had its id truncated at a random position, an
intermittent auth failure reproducing only for SOME keys. The id is hex now, and
the guard is falsified: reverting the alphabet reds both
The_generated_key_id_never_contains_the_separator and, as a bonus signal,
Two_generated_keys_never_collide, because truncated ids collide.
ApiKeyId is NOT NULL, unlike the nullable NodeAclId precedent whose filtered
unique index would admit two NULL ids. Fine for a config row, not for a
credentials lookup key.
OpcUaServer.csproj now references Core. Until now it did not, so
IPermissionEvaluator was literally uncallable from the node manager — the place
enforcement has to happen. No new transitive package surface.
Schema doc updated for both, and its NodeAcl block flagged for pre-existing drift
(it still documents the GenerationId column v3 dropped) without fixing it here.
Core 247, OpcUaServer 364, Configuration 128 — all green.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
Stage 1a of the API-key ACL design. Behaviour-neutral: the trie walker already
matched an opaque string, so only what produces that string changes. No
enforcement is wired by this commit and none of the two live role gates move.
Renamed through the ACL model: NodeAcl.Principal, TrieGrant.Principal,
MatchedGrant.Principal, AuthorizationDecision.Principal,
UserAuthorizationState.Principals, and PermissionTrie.CollectMatches's parameter.
LdapGroupRoleMapping is a DIFFERENT entity that also has an LdapGroup column and
is deliberately untouched — it backs AdminUI login, which keeps LDAP.
The migration is the interesting part. EF scaffolds RenameColumn/RenameIndex, so
rows survive — but it cannot know that dbo.sp_ComputeGenerationDiff checksums
NodeAcl.LdapGroup. That proc is created in V3Initial.StoredProcedures and never
redefined since, and SQL Server does not validate a procedure body when a column
is renamed out from under it: the object keeps reporting healthy and fails at
CALL time with Invalid column name, silently breaking config-drift detection.
Verified against the live dev DB that the deployed proc really does reference the
column. The migration therefore re-emits the proc, as one parameterised template
shared by Up and Down so the two cannot drift.
Prose that describes what is enforced TODAY (fleet-wide LDAP group -> role) is
left alone, because it is still accurate until stage 2 lands.
UserAuthorizationState's refresh machinery is retained unchanged but now carries
a note that it is LDAP-shaped and largely vestigial here: an API-key principal
set cannot change out of band, since keys and ACL rows both ride the deployment
artifact, so the generation stamp rather than the freshness timer is the real
invalidation signal.
41 authorization tests pass; solution builds clean.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
The sibling of #521's DriverHostStatus, dead for exactly the same reason. Before this change the
only reference in the entire repo was the DriverInstanceResilienceStatuses DbSet declaration: no
writer, no reader, and the table empty on every deployment.
Its entity doc described two components that were never built — a HostedService sampling the
tracker every ~5 s into this table, and an AdminUI that left-joined it on DriverHostStatus for
display. There is no sampler, that join exists in no razor file, and DriverHostStatus itself was
removed in #521.
Per-cluster mesh Phase 4 also makes the design unbuildable now: AddOtOpcUaConfigDb is gated on the
admin role, so a driver-only node has no ConfigDb connection to write rows with. Any per-node
runtime-status table has this problem.
The live data was never missing — it just does not go through SQL. Resilience counters reach the
AdminUI's DriverStatusPanel as DriverResilienceStatusChanged over the driver-resilience-status
telemetry channel into the in-memory IDriverResilienceStatusStore. Verified consumed: the panel
injects the store, fed by both DriverResilienceStatusBridge (DPS) and TelemetryDialSupervisor
(gRPC).
If breaker HISTORY across a central restart is ever wanted — the one thing the last-value-only
store cannot give — it needs a CENTRAL-side writer fed by that stream, never a driver-node writer.
Keeping an empty table now bought nothing toward that.
Migration DropDriverInstanceResilienceStatusTable; the scaffolder's data-loss warning cannot bite
on a table nothing ever inserted into, and Down() recreates it exactly.
Also corrected: DriverResilienceStatusTracker's doc-comment claimed CurrentInFlight fed
DriverInstanceResilienceStatus.CurrentBulkheadDepth "until the full telemetry observer lands". The
observer landed in Phase 5; the table never had a writer.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
`IHostConnectivityProbe` was a dead surface: eleven drivers implement it, `GetHostStatuses()`
had ZERO production call sites, and `OnHostStatusChanged` had no subscriber outside the Galaxy
driver's own aggregator. Per-host connectivity was computed by every driver and read by nobody.
The issue offered "build the publisher or delete it". The publisher as its entity doc described
it — driver nodes upserting `DriverHostStatus` rows — is not buildable: per-cluster mesh Phase 4
gates `AddOtOpcUaConfigDb` on the `admin` role, so a driver-only node has no ConfigDb connection
to write rows with. So the capability is kept and the transport changed.
`DriverHealthChanged.HostStatuses` now carries the probe result to `/hosts` as a Hosts column.
That channel already reached the page, already survives the mesh split via the Phase 5 gRPC
telemetry stream, and already replays a last-value snapshot on re-subscribe — so per-host state
re-primes after a reconnect without a durable store. Both halves of the interface finally do
what they are for: the event triggers a prompt publish, the pull is the source of truth.
The point of the column is the case the driver-level state chip structurally cannot express: a
multi-device driver stays aggregate-Healthy while ONE of its devices is unreachable.
Two traps, both pinned by tests that were falsified against the prod code:
- The host digest MUST be in the publish fingerprint. On a single-host-down transition every
other fingerprint component is unchanged, so the dedup would swallow exactly the publish
carrying the news — the trap that already bit the rediscovery signal. Removing it turns the
guard test red, verified.
- null (no probe) must stay distinct from empty (probe with no hosts). proto3 cannot tell an
absent repeated field from an empty one, hence the explicit `has_host_statuses` flag;
collapsing them would render every probe-less driver as one whose devices are all fine.
Dropped: the DriverHostStatus entity, enum, DbSet, model config and table (migration
DropDriverHostStatusTable — empty on every deployment, so the scaffolder's data-loss warning is
moot, and Down() recreates it exactly).
Found en route, NOT fixed here: `DriverInstanceResilienceStatus` is the identical defect — no
writer, no reader, only a DbSet declaration, while the live data rides the
`driver-resilience-status` telemetry channel. Its doc-comment now states that rather than
describing the sampler and AdminUI join that were never built. Filed as #524 rather than widening
this schema change beyond what was asked.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
Scripted-alarm condition state lives in LocalDb (Phase 4); the ConfigDb-backed
Ef store + ScriptedAlarmState table are now dead. Removes the test-harness Ef
fallback (a DB-backed node with no store now skips the alarm host), deletes the
store + entity + model config, and drops the table via migration.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 1 gate step 4 failed: setting Enabled = 0 to take a node out of service
returns 422 ClusterEnabledNodeCountMismatch, because
DraftValidator.ValidateClusterTopology requires the enabled-node count to equal
ServerCluster.NodeCount. On a Warm/Hot pair — every cluster on the rig, and
every cluster in the target topology — Enabled can therefore never be the
maintenance hatch. The plan called step 4 "the check that makes step 3
acceptable to ship", so Task 3's behaviour change was not shippable as it stood:
a node down for maintenance would block every deployment to its cluster.
The two rules were each reasonable and contradictory together — the validator
reads Enabled as "part of the declared topology", Phase 1 additionally read it
as "expect an ack". Split the meanings rather than weaken either rule:
Enabled part of the declared topology (validator, untouched)
MaintenanceMode expected to participate now (coordinator + reconciler)
Rejected alternatives: counting configured rather than enabled nodes (drops the
guard against booting a pair into InvalidTopology); downgrading the rule to a
warning (weakens a deploy gate for everyone); shipping with no hatch.
Sabotage: dropping !n.MaintenanceMode from the coordinator query turns the new
test red. It also asserts both nodes remain Enabled, so the fix cannot quietly
regress to disabling the row after all.
Configuration.Tests 95/95, ControlPlane.Tests 101/101, solution builds clean.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Per-cluster mesh Phase 1 groundwork. Once the meshes split (Phase 2) central
can no longer see a site node's appsettings, so the transport ports it must
dial have to live in a row central can read.
AkkaPort is non-nullable with a 4053 default — every node listens on a
remoting port, so 0 is never a truthful value and pre-existing rows must
migrate to something real. GrpcPort is nullable with no default: nothing
listens on it until Phase 5, and a non-null default would assert a port that
does not exist.
Both are documented as central's dial targets rather than the node's own
binding config; the duplication against Cluster:Port is reconciled in Task 4.
The new SchemaCompliance test is red until the migration lands (Task 2).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Adds <summary>/<param>/<returns>/<inheritdoc> where missing and removes
project bookkeeping IDs (task/tracking refs) from shipped code comments,
so the docs read cleanly and CommentChecker is quiet except for known
false positives (PLC/protocol terms, event/IEqualityComparer inheritdoc).
Doc/comment-only; no logic changed; solution builds clean.
Replace "SystemPlatform mirror tag", "Galaxy alias", and "SystemPlatform-kind" in doc-comments and
test names with neutral accurate wording ("FolderPath-scoped tag", "EquipmentId == null", etc.).
No code, logic, or test bodies changed — comments and one test method name only.
Resolves the code-review notes on 95be607a + the AdminUI bundle: the
EnsureVariable docs (IOpcUaAddressSpaceSink, OtOpcUaNodeManager) and the Tag
entity doc no longer say 'Galaxy / SystemPlatform / alias'; the DriverHostActor
ForwardToMux comment now states the real equipment-tag value-routing gap (the
FullName→NodeId 'live values' milestone) instead of claiming Galaxy values map
straight through.
Persist the canonical AuditOutcome and make structured audit rows visible.
- ConfigAuditLog gains a nullable Outcome column, stored as the AuditOutcome
enum member name (nvarchar(16), mirroring how AdminRole is persisted). The
AuditWriterActor flush now writes Outcome = evt.Outcome.ToString(). Nullable so
legacy rows and the bespoke stored-procedure path (no derived outcome) write
NULL.
- Migration 20260602135350_AddConfigAuditLogOutcome: additive nullable column,
no backfill. Up adds the column, Down drops it. Chains after
20260602112419_CanonicalizeAdminRoles; `dotnet ef migrations
has-pending-model-changes` is clean.
- ClusterAudit visibility fix: the page filtered solely on ClusterId, but the
structured AuditWriterActor path stamps NodeId (ClusterId null), so those rows
were invisible. Extracted ClusterAuditQuery.ForClusterAsync (shared by the page
and tests) which ORs in rows whose NodeId belongs to a node in the cluster —
membership resolved from ClusterNode (NodeId -> ClusterId). SP-path
ClusterId-stamped rows still match.
Tests: ControlPlane 45/45 (adds Outcome persistence + Denied-outcome asserts);
new Configuration ClusterAuditQueryTests 3/3 (both-paths visible, other-cluster
excluded, page-size cap); AdminUI 121/121. Configuration Unit suite is green on a
clean run (a pre-existing timing flake in ResilientConfigReaderTests, untouched
here, occasionally fails under parallel load and passes in isolation).
Adds <summary>, <param>, <typeparam>, and <inheritdoc/> tags to public
members surfaced by commentchecker — resolves 5,847 of 5,869 issues
(99.6%) across three /fixdocs passes.
ConfigAuditLog gains two nullable columns (EventId, CorrelationId) + a filtered
unique index UX_ConfigAuditLog_EventId. EF migration
20260526105027_AddConfigAuditLogEventIdColumns is additive (nullable + filtered
index = legacy rows backfill cleanly).
AuditWriterActor now writes EventId + CorrelationId into the dedicated columns
instead of synthesising a JSON wrapper into DetailsJson. Cross-restart dedup
is now real: a retry of an already-flushed batch hits the unique index and
SaveChanges throws; the existing catch drops the duplicate without losing the
rest of the batch.
WrapDetails helper deleted — F4 (its JSON hardening) becomes moot.
AuditWriterActorTests.Details_wrapper_embeds_eventId_and_correlationId renamed
+ rewritten to assert against the columns. All 29 ControlPlane tests pass,
all 95 v2 tests green.
Phase 1e of the v2 entity-model rewrite. With the FKs gone (Task 14b) and
the apply pipeline replaced (Task 14c), the v1 draft/publish entities have
no remaining v2 consumers.
Deleted entity classes:
src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Entities/ConfigGeneration.cs
src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Entities/ClusterNodeGenerationState.cs
Deleted enum classes (no v2 consumers):
src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Enums/GenerationStatus.cs
src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Enums/NodeApplyStatus.cs
OtOpcUaConfigDbContext changes:
- Removed DbSet<ConfigGeneration> ConfigGenerations
- Removed DbSet<ClusterNodeGenerationState> ClusterNodeGenerationStates
- Removed ConfigureConfigGeneration(modelBuilder) call + method body
- Removed ConfigureClusterNodeGenerationState(modelBuilder) call + body
- Tidied the "v2 deploy-model tables" header comment
Navigation property cleanup:
- ServerCluster.Generations collection -> removed
- ClusterNode.GenerationState navigation -> removed
doc-comment cref cleanup (replaced <see cref="X"/> with <c>X</c> for the
deleted types so the C# XML comment compiler doesn't fail with CS1574):
- Deployment.cs (cref to ConfigGeneration)
- NodeDeploymentState.cs (cref to ClusterNodeGenerationState)
- Core/OpcUa/EquipmentNodeWalker.cs (cref to ConfigGeneration in the
EquipmentNamespaceContent record's doc-comment; while there, removed
"All four collections are scoped to the same ConfigGeneration" since
that's no longer true in v2)
Verification:
src/Core/ZB.MOM.WW.OtOpcUa.Configuration -> 0 errors
src/Core/ZB.MOM.WW.OtOpcUa.Core -> 0 errors
tests/Core/ZB.MOM.WW.OtOpcUa.Configuration.Tests -> 0 errors
tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests -> 0 errors
whole solution -> 15 errors
(all in Server/Admin; transitive Server.Tests/Admin.Tests skip per the
parent's failure, so the per-project count dropped vs Task 14d's 71)
Phase 1d of the v2 entity-model rewrite. The static RedundancyRole column
is replaced by Akka cluster's role-leader-of-"driver" election at runtime
(see RedundancyStateActor + ServiceLevelCalculator in Task 35).
Changes:
- Removed `public required RedundancyRole RedundancyRole` from
ClusterNode entity.
- Removed `e.Property(x => x.RedundancyRole).HasConversion<string>()...`
mapping from OtOpcUaConfigDbContext.ConfigureClusterNode.
- Removed the `UX_ClusterNode_Primary_Per_Cluster` filtered unique index
(filter referenced [RedundancyRole]='Primary').
- Dropped `using ZB.MOM.WW.OtOpcUa.Configuration.Enums` from ClusterNode.cs
(no longer needed).
- Deleted `Enums/RedundancyRole.cs` — the enum is unused in v2-kept code.
- DraftValidator: dropped the "exactly one Primary per cluster"
validation block. Comment in place explaining v2 picks primary at
runtime via Akka.
- DraftValidatorTests: dropped ValidateClusterTopology_flags_multiple_Primary
test; reworked BuildNode helper to no longer take a `role` argument.
Untouched (Server + Admin still reference RedundancyRole; accepted broken
per Task 56 policy):
src/Server/ZB.MOM.WW.OtOpcUa.Server/Redundancy/{ClusterTopologyLoader,
RedundancyStatePublisher, RedundancyTopology, ServiceLevelCalculator}.cs
src/Server/ZB.MOM.WW.OtOpcUa.Admin/Services/RedundancyMetrics.cs
DB-runtime tests will fail against the new schema (Task 14f's migration
drops the column) — to be updated in Task 14f's SchemaComplianceTests
update:
- SchemaComplianceTests.cs:55 (expected filtered index list)
- StoredProceduresTests.cs:263 (raw INSERT names the column)
Verification:
src/Core/ZB.MOM.WW.OtOpcUa.Configuration -> 0 errors
tests/Core/ZB.MOM.WW.OtOpcUa.Configuration.Tests -> 0 errors
whole solution -> 71 errors
(70 from Task 14b in Server/Admin, +1 new Server/Redundancy reference)
Phase 1a of the v2 entity-model rewrite. Adds:
public byte[] RowVersion { get; set; } = Array.Empty<byte>();
and the EF Core mapping
e.Property(x => x.RowVersion).IsRowVersion();
to 12 live-edit entities:
Equipment, DriverInstance, Device, Tag, PollGroup, Namespace,
UnsArea, UnsLine, NodeAcl, Script, VirtualTag, ScriptedAlarm
These are the entities that v2 admins will edit directly via
AdminOperationsActor (no draft staging). RowVersion enables
last-write-wins detection when two operators race on the same row.
GenerationId FKs are still in place on these entities (removed in Task 14b);
this commit only adds the rowversion column so the migration in Task 14f can
emit ADD COLUMN before DROP FK as a single atomic step.
Phase 1 entities for the v2 live-edit + snapshot-deploy model:
Deployment — immutable artifact snapshot (replaces v1 ConfigGeneration row)
Status enum {Dispatching, AwaitingApplyAcks, Sealed,
PartiallyFailed, TimedOut}; carries the SHA256 RevisionHash and
the SnapshotAndFlatten() ArtifactBlob; RowVersion for optimistic
concurrency.
NodeDeploymentState — per-(node, deployment) apply progress row owned by
DriverHostActor (replaces single-row ClusterNodeGenerationState).
Composite key (NodeId, DeploymentId) gives the
ConfigPublishCoordinator the full history it needs to
reconstruct in-flight state after a failover.
ConfigEdit — append-only audit row written by AdminOperationsActor on every
mutating op; optional ExecutionId correlates edits inside one
admin transaction (e.g. an import batch).
DataProtectionKey — ASP.NET DataProtection key ring storage via
IDataProtectionKeyContext so every admin-role node decrypts
the same cookies without sharing a filesystem.
OtOpcUaConfigDbContext now implements IDataProtectionKeyContext and registers four new
DbSets + four new ConfigureXxx mappings.
Central package bumps (forced by Microsoft.AspNetCore.DataProtection.EntityFrameworkCore
10.0.7's transitive dep):
Microsoft.EntityFrameworkCore.{,Design,InMemory,SqlServer} 10.0.0 -> 10.0.7
Microsoft.Extensions.{Configuration.Abstractions,Configuration.Json,Hosting,Hosting.WindowsServices,Http} 10.0.0 -> 10.0.7
EF migration generation + the ConfigGeneration drop + RedundancyRole column removal are
deferred to Task 14 (high-risk, non-parallelizable).
Group all 69 projects into category subfolders under src/ and tests/ so the
Rider Solution Explorer mirrors the module structure. Folders: Core, Server,
Drivers (with a nested Driver CLIs subfolder), Client, Tooling.
- Move every project folder on disk with git mv (history preserved as renames).
- Recompute relative paths in 57 .csproj files: cross-category ProjectReferences,
the lib/ HintPath+None refs in Driver.Historian.Wonderware, and the external
mxaccessgw refs in Driver.Galaxy and its test project.
- Rebuild ZB.MOM.WW.OtOpcUa.slnx with nested solution folders.
- Re-prefix project paths in functional scripts (e2e, compliance, smoke SQL,
integration, install).
Build green (0 errors); unit tests pass. Docs left for a separate pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Joseph Doherty