Baselining one table used to set a single per-DATABASE needs_snapshot bit, and a
snapshot streams every registered table in full — so opting one table into
replication late re-sent all of them, in both directions at once on a pair
enabling replication late. On a database holding a high-volume table (an event
log at its retention cap) that is the largest transfer this library performs.
0.2.1 documented the cost and deferred the fix as needing an on-disk change.
Schema v3 adds __localdb_snapshot_state (table_name PK, needs_snapshot), created
by the shared DDL with IF NOT EXISTS so an older file gains it on open; the v2->v3
transition is a version stamp only. RegisterReplicated(..., baselineExistingRows:
true) now flags that table alone. The database-wide flag survives unchanged for
the paths that genuinely reach every table — a cap prune or an impossible peer ack
loses delta continuity for the oplog as a whole. DeregisterReplicated drops a
table's pending flag with its ledger rows, since nothing would ever stream (and so
clear) it afterwards.
Wire compatibility is additive and negotiated by CAPABILITY, not version:
lib_schema_version stays 1 because ValidateHandshake compares it fail-closed for
equality, so bumping it would stop 0.1.x/0.2.x pairs syncing outright. Handshake
gains supports_partial_snapshot (field 5) and SnapshotBegin gains a tables scope
(field 2) where EMPTY means all tables — precisely what a pre-0.3.0 sender emits
and what a pre-0.3.0 receiver assumes. A peer that does not advertise the
capability is served a full snapshot: a scoped one would make it advance
last_applied_remote_seq to the snapshot's as-of and silently discard every delta
below it for the tables left out.
For the same reason a partial snapshot moves no watermark on either side. The
receiver keeps last_applied_remote_seq (carrying only the HLC across, which is
MAX-guarded), and the sender keeps _sentThruSeq, so the deltas the snapshot
overlaps are still pumped and merged by LWW.
Crash safety: snapshot debt now clears on the receiver's new SnapshotAck rather
than on enqueue, so an ungraceful exit part-way through re-streams the same scope
on reconnect. The ack rides the unbounded CONTROL lane like DeltaAck, so it
composes with 0.2.1's bounded 64-message inbox and its out-of-handshake snapshot.
Against a pre-0.3.0 peer the 0.2.x clear-on-enqueue behaviour is kept — there is
no ack to wait for — with the handshake's horizon gap check as the net.
Tests 199/199 (was 188): single-table baseline streams only that table, mixed-
version interop (peer without the capability gets a full snapshot), debt standing
until SnapshotAck, partial snapshot not advancing the receiver's watermark, no
SnapshotAck emitted to an old peer, end-to-end over two real databases, plus v2->v3
upgrade and per-table flag isolation.
README, ILocalDb remarks (which named this as the follow-up) and the umbrella
CLAUDE.md index updated.
The umbrella LocalDb row and the pack/test quick-reference now name 0.2.1 (188 tests,
3 nupkgs on the feed). Records both findings, the snapshot-out-of-handshake change that
bounding the inbox required, and the deferred per-table needs_snapshot. Consumer pins are
unchanged: ScadaBridge still pins 0.2.0, OtOpcUa still pins 0.1.3.
deployment.md:
- §1/§2.1: add wonder-app-vq03, wonder-app-vq04 and wonder-sql-vq03 with IPs and
per-host roles; new §7.1 QA port map + live active/standby state.
- QA is a real 2-node HA pair (ScadaBridge central+site pairs, OtOpcUa pair) where
DEV is a single all-in-one box, so it is the only environment here that exercises
failover and leader election. MxGateway/HistorianGateway/Overview run on vq03 only.
- Every QA fact is marked scan (port probe) or dash (read off the QA Overview
dashboard's live polling). No QA config file was read; connection strings and
service names stay unverified.
- Correct the tier: vd03 is DEV, not production (vd=dev, vq=qa, per
mes-delmia-integration-api.md §1.1). Its ASPNETCORE_ENVIRONMENT really is
"Production" -- a .NET overlay name, orthogonal to the tier, and the likely source
of the original error. No production box has been identified.
- Record that QA requires Basic auth on /management where DEV does not, so DEV-only
procedures relying on unauthenticated management calls will not work on QA.
- Record that inbound API keys never cross environments (per-stack peppered-HMAC,
excluded from config bundles) plus the 401-vs-400 probe, with the caveat that auth
runs before route resolution so a 401 needs a positive control.
- Withdraw the SSH-on-2222 claim made earlier in this same pass: all three wonder
boxes accept TCP on 2222 and send no SSH banner, so servecli is not serving SSH and
no shell access was available. Port scan cannot establish a service; banner grab can.
CLAUDE.md:
- Overview: record the second instance on wonder-app-vq03:5320, and that its real-FQDN
registry is correct rather than a divergence -- the 127.0.0.1 rule is specific to
vd03 co-locating every instance on one host.
- Add the DEV-vs-prod note pointing at deployment.md §2.1/§7.1.
CLAUDE.md: feed now carries 0.2.1; record the referral defect and that every
app is still exposed until it bumps off 0.1.5 (a functional fix, not
housekeeping), plus the new tools/LdapProbe.
ldap_setup.md: findings from probing the live NAM directory --
- LDAPS 636, StartTLS 389 and plain 389 all connect on both DCs. The only TLS
chain error is UntrustedRoot, so installing zimmer-rootca is sufficient; the
RSA 1024-bit key is rejected only by OpenSSL at SECLEVEL=2, i.e. Debian .NET
containers, not by macOS or Windows hosts.
- The UG_WW_* Wonderware group convention, 39 groups enumerated directory-wide.
This partly overturns this doc's earlier "no group->role convention exists"
claim: that was true of Ignition, which was the only source audited, and
false of the domain. Three shapes with a consistent role suffix (ADMIN, RO,
OPERATOR, ENGINEER, SUPERVISOR, TESTER) across environment and site axes.
- A proposed mapping onto CANONICAL-ROLES, with the three gaps left open
rather than guessed: SUPERVISOR/TESTER have no canonical equivalent, nothing
maps to Deployer, and the DEV/QA/PROD axis is orthogonal to role so it must
ride in the per-project scope payload or the role set triples.
- The referral defect and its fix.
This repo hosts one application besides its shared libraries, and the index did
not mention it at all. Adds what it is, that its registry ships empty by design,
and the two constraints that bit during the first real deployment: publish
self-contained (wonder has no .NET 10 runtime), and do not put co-located
instances on a shared host name, because leader-to-instance resolution matches on
host only and the first match wins the chip for every group.
CLAUDE.md (Secrets row):
- mxgw is NOT Bootstrap-free (libman.json pins bootstrap@5.3.3, App.razor:7
links it ahead of ThemeHead). The scadaproj#2 modal gate had skipped mxgw on
that false premise; gate since run and PASS 2026-08-11.
- scoped-CSS bundle claim scoped to its 2026-07-19 date; OtOpcUa links one now
too, so it is two of four rather than one.
admin_ui_cleanup.md — five rounds now (adds mxgw, HistorianGateway, ScadaBridge):
- 1a-1: test sources are a fifth definition site, checked BEFORE the ghost list.
A class matched only by a test is a selector, never a ghost. Extends to
substring and visible-text assertions, so button labels are load-bearing.
- 1a-4: two inverted button-sizing instances (literal font-size neutering
btn-sm app-wide; a chassis class standing in for the button system).
- 1a-5: determine dark scheme per app; where one exists the step inverts.
- 1a-6 (new): a token override emitted as an inline style on the shell root
defeats every scheme-level :root override. OtOpcUa carries the inert-but-
latent instance.
- 1c: reword nothing before checking test sources; the docs may be the wrong
document, not the page.
- 2: btn-sm fold breaks button.btn-*.btn-sm locators; cell-clip is wrong for
resizable columns; briefs carry the rubric, not just line numbers.
- verification: baseline pre-existing failures by measuring; a suspiciously
fast green is a vacuous pass, not a pass.
- live gate: mandatory bUnit or not; hit-testability over visibility; measure
behaviour where computed values mislead (-webkit-box serializes flow-root);
record what could not be exercised.
- new: exemptions from a family-wide sweep must re-derive their premise.
Program closeout: scadaproj#4 CLOSED (comment issuecomment-5751) — ScadaBridge's
central pair now runs ONE shared SQL-Server store in Grpc mode (0.5.0
FallbackEndpoints, 0.5.1 virgin-DB migrator race fix; ScadaBridge @ e697477c),
live-gated PASS 5/5 + re-drill. The pre-Serilog container-wedge defect the gate
reproduced twice is filed as ScadaBridge#34 (pre-existing, out of program scope).
- clustered-secrets.md: central pair = one shared SQL store (connstr literal/env,
never ${secret:}); rewrap = ONE central run + both nodes of each site pair;
FallbackEndpoints same-store-only rule; 0.5.1 minimum for central pairs
- components/secrets/GAPS.md: resolution section appended; stale wait-on-#4 struck
- CLAUDE.md Secrets row: #4 CLOSED, family 0.5.1, gate + residual recorded
- tasks.json: all 6 tasks completed
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Task 9 (closeout) of the Secrets gRPC-hub program. Bookkeeping only; no code.
- clustered-secrets.md, Production enablement: the gRPC hub is no longer
pending. It is BUILT, SHIPPED (0.4.1) and LIVE-GATED PASS 4/4 (ScadaBridge
main @ f6c3f7c5, docs/plans/2026-08-07-secrets-grpc-hub-live-gate.md), so the
"until that ships, ScadaBridge replication stays off" line is replaced with
the truth: the code path is merged and gated, and production enablement is a
separate operator action behind a still-default-off flag. OtOpcUa's
pair-local topology is recorded as live-gated (lmxopcua master @ 1bd79c8e)
rather than pending re-gate. Rotation fan-out corrected: on ScadaBridge it is
BOTH central nodes plus both nodes of every site pair, because each central
node has its own independent store.
- components/secrets/GAPS.md: new "gRPC hub delivered" section recording the
0.4.0/0.4.1 publish, both gates with SHAs and gate-doc paths, and the
residual. Existing narrative left intact and amended, not rewritten.
- CLAUDE.md Secrets row: lib version 0.3.0 -> 0.4.1 and the app-pin claim
corrected (ScadaBridge pins 0.4.1 for the Grpc replicator; the other three
still pin 0.2.3); the "production topologies DECIDED" region now records
delivery, both gates, default-off, and the residual.
- Plan ledger: task 9 completed.
Residual carried forward as scadaproj#4 — ScadaBridge's central pair does not
converge with itself (central-b served an authenticated but EMPTY manifest all
gate while central-a held every secret) and the site client dials a single
endpoint; these are one design question, since failover to an empty hub would
succeed and silently stop convergence. scadaproj#3 is closed.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Design doc + runbook Production-enablement section + GAPS/index updates.
The previously-untracked production-KEK decision is now recorded: one
fleet-wide KEK via appsettings/env (never committed); ScadaBridge rejects
the SqlServer hub on topology grounds (sites talk to central, never the
DB) in favor of a new ZB.MOM.WW.Secrets.Replicator.Grpc package —
pull-only, updates originate at central only, by design.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
The InitJoin self-form watchdog (a timer calling Cluster.Join(SelfAddress) after a
window) was implemented and REJECTED after live testing: it sits outside Akka's join
handshake, so on a routine standby restart — peer alive, join stalled behind removal of
the node's own stale incarnation — it fires mid-join, abandons the in-flight join and
forms a permanent second cluster at the same address (measured: still split after 90s;
OtOpcUa hit it live in its manual-failover drill).
The safety argument 'a booting node only self-forms when no seed answers InitJoin' is
true of Akka's NATIVE first-seed rule (FirstSeedNodeProcess, inside the handshake) and
FALSE of the watchdog, which can fire while a seed is answering. That reasoning error is
now called out explicitly so no future plan re-derives the pattern.
- docs/plans/2026-07-22-initjoin-selfform-fallback.md: dated correction banner (history
preserved) + the two surviving false claims struck in place (behavior-spec row and the
Mechanism section's 'races are benign'); points at ScadaBridge
Component-ClusterInfrastructure.md as the surviving design.
- akka_failover.md 6.1: shipped design stated as self-first seed ordering plus the
opt-in, default-OFF simultaneous-cold-start bootstrap guard; added a
never-re-derive-the-watchdog note. 6.2: two items my 6.1 edit contradicted are marked
closed (OtOpcUa auto-down live gate, mesh election scope — both done 2026-07-24).
- CLAUDE.md: ScadaBridge issue #14 is CLOSED (2026-07-24), not open — v3 dual-namespace
live gates all passed, ScadaBridge consumes OtOpcUa v3 end-to-end.
#520 stage 6 (master 2026-08-01, live-gated). With Security:Acl:Enabled ON,
every data-plane operation needs a matching NodeAcl grant, not just writes /
alarm acks / HistoryRead. The stage-5 entry told ScadaBridge that a read-only
consumer was 'unaffected either way' — that now holds only while the flag is
off, which is the default.
Also records the two stage-6 behaviours a consumer would otherwise discover the
hard way: a deep Browse grant makes its ancestor folders browsable (so it
exposes their names), and revoking a grant stops a live subscription — one
BadUserAccessDenied then silence, resuming by itself if the grant returns, with
no reconnect needed.
Default is off, so a read-only address-space consumer is unaffected. With it on,
writes / alarm acks / HistoryRead need a matching grant for the principal the
session authenticates as — anonymous unless ScadaBridge is issued a key.
Reads, browses and subscriptions stay ungated in both modes.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
Joseph Doherty