Tasks 5 and 6 of the Phase 2 plan, committed together because their test
fallout is entangled — several fixtures construct both stores.
StoreAndForwardStorage and SiteStorageService now take ILocalDb. Connections
come from ILocalDb.CreateConnection(), which hands out an already-open,
pragma-configured connection carrying the zb_hlc_next() UDF the capture triggers
call; a raw connection would lack the UDF and every write to a replicated table
would fail closed. Deleted with the connection strings: S&F's
EnsureDatabaseDirectoryExists and its per-open busy_timeout pragma, and the site
service's BusyTimeoutFloorSeconds normalization — LocalDb owns all of it now.
DI: AddSiteRuntime's string overload is gone (nothing left to supply), so the
Host calls the no-arg form. ScadaBridge:Database:SiteDbPath and
StoreAndForwardOptions.SqliteDbPath survive only as the migrator's source
locations in Tasks 8/9.
Two things the plan did not anticipate, both worth reading:
1. FOUND A REAL LATENT DEFECT, from Phase 1, now fixed. The plan assumed
directory creation simply moved to LocalDb along with file ownership. It did
not: the LocalDb library never creates the parent directory, and
SqliteLocalDb opens the file eagerly in its constructor — so a missing
directory is a hard boot failure ("SQLite Error 14: unable to open database
file"), not a degraded start. The default site config points at the RELATIVE
path ./data/site-localdb.db, so any site node without a pre-existing data/
directory fails to boot. The docker rig escapes only because its volume mount
happens to create /app/data — a coincidence that would have hidden this until
a bare-metal or fresh deployment. This has been latent since Phase 1 made
LocalDb:Path required; deleting S&F's EnsureDatabaseDirectoryExists here
would have widened it. Re-established the guarantee at the layer that now
owns the path (SiteLocalDbDirectory.Ensure, called before AddZbLocalDb) and
pinned it with SiteLocalDbDirectoryTests. Non-vacuity is not assumed: two
tests written against the wrong assumption failed with exactly this
SQLite Error 14 before the fix existed.
2. Test fallout was ~7x the plan's estimate. The plan named "fixtures" in one
project; the constructor change actually reaches 40 files across 7 test
projects, and most used Mode=Memory;Cache=Shared — which LocalDb has no
equivalent for, so every one had to move to a real temp file. Rather than
copy the Phase 1 TestLocalDb fixture into 7 projects, added a shared
tests/ZB.MOM.WW.ScadaBridge.TestSupport library (not a test project) so the
WAL-sidecar cleanup and the "real, not stubbed" rationale live in one place.
Retargeted rather than deleted, in both directions: the S&F WAL test now asserts
against the LocalDb-backed store (WAL genuinely is LocalDb's job), while the
directory-creation test moved to Host.Tests (that guarantee is NOT LocalDb's).
SiteStorageServiceTests.Initialize_EnablesWalJournalMode got the same treatment.
DeploymentManagerMediumFindingsTests induced a persistence failure via an
unopenable path, which no longer reaches the assertion since the fixture now
throws first; it induces the same failure shape via an uninitialized store.
Verified: full solution build 0 warnings; SiteRuntime 532, Host 318,
AuditLog 355, ExternalSystemGateway 142, HealthMonitoring 97,
StoreAndForward 153 — 1597 passed, 0 failed.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Task 4 of the Phase 2 plan. All nine table definitions plus MigrateSchemaAsync
and TryAddColumnAsync move into a static sync Apply(SqliteConnection) depending
only on Microsoft.Data.Sqlite, so the Host can apply the DDL to a LocalDb-managed
connection before RegisterReplicated installs the capture triggers.
Per the plan, PRAGMA journal_mode=WAL stays in InitializeAsync — LocalDb owns
the connection's pragmas, and the service still opens its own connections until
Task 6.
One deviation: TryAddColumnAsync's `catch (SqliteException) when
(ex.Message.Contains("duplicate column"))` becomes a PRAGMA table_info probe,
matching OperationTrackingSchema. The message-matching form depends on an error
string that is not part of SQLite's contract, and it swallowed every
SqliteException whose message happened to contain that substring. The probe also
drops the ILogger dependency, which is what let the class become static. Cost:
the per-column "Migrated: added column" info log is gone — it fired once per
column per legacy database and nothing consumes it.
Also added a test beyond the plan's specified one, for the same reason as Task 3:
the specified test asserts against a freshly-created database, where CREATE TABLE
already lists the migration columns, so it would pass with every ALTER deleted.
The added test starts from the pre-migration shapes with a row present and proves
the migration path runs and preserves data across a NOT NULL DEFAULT add.
SiteRuntime suite 532 passed / 0 failed; full solution build 0 warnings.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Task 4 of the LocalDb Phase 1 adoption plan.
OperationTrackingStore took IOptions<OperationTrackingOptions> and opened its own
SqliteConnection from ConnectionString. It now takes ILocalDb and gets every
connection - writer and the two ad-hoc reader paths - from CreateConnection().
This is not cosmetic. OperationTracking is a RegisterReplicated table, so its
capture triggers call zb_hlc_next(). That UDF is registered per connection by
ILocalDb and by nothing else, so a raw SqliteConnection would fail closed on
every write. Connections from CreateConnection() also arrive already open -
calling Open()/OpenAsync() on one throws - hence the removed OpenAsync calls on
the reader paths.
OperationTrackingOptions.ConnectionString is now vestigial for this store; the
database location is LocalDb:Path. The options class stays (retention settings)
and the config key stays bound for the site config DB.
InitializeSchema is kept but is now always a no-op in the host: onReady runs
while ILocalDb is being constructed, strictly before this constructor can
receive it. It remains so a directly-constructed store (tests, tooling) is
self-sufficient.
Tests: the store fixture moves off mode=memory&cache=shared onto a real ILocalDb
over a temp file. There is no in-memory mode - LocalDbOptions.Path is a
filesystem path - and testing through a raw in-memory connection would no longer
resemble the host. Verifier connections stay raw on purpose: this fixture never
registers the table, so no trigger fires and the UDF is never reached.
Three DI fixtures needed LocalDb:Path added, because resolving
IOperationTrackingStore now forces ILocalDb construction:
SiteCompositionRootTests, SiteAuditWiringTests, and the AuditLog
CombinedTelemetryHarness.
Verified: build 0 warnings; Host 294/294, SiteRuntime 530/530, AuditLog 354/354.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Tasks 2 + 5a-writer + 5b of the LocalDb Phase 1 plan, landed together because
they are one indivisible change: the schema, the writer that fills it, and every
consumer that assumed the old id semantics.
WHY the id changes. Site pairs will replicate site_events with last-writer-wins on
the primary key. With INTEGER PRIMARY KEY AUTOINCREMENT both nodes independently
mint 1, 2, 3... for unrelated events, so sync would treat them as the same row and
silently overwrite. A GUID makes the event log a pure union across the pair.
Task 2 - schema extracted so the Host's AddZbLocalDb onReady can create the tables
before RegisterReplicated installs capture triggers (pre-registration rows are
never captured):
- new OperationTrackingSchema.Apply / SiteEventLogSchema.Apply, plain
Microsoft.Data.Sqlite, no LocalDb dependency
- both stores delegate their InitializeSchema to them (idempotent, so a
directly-constructed store still works)
- OperationTracking is unchanged - it already had a TEXT PK and replicates as-is
Task 5a - SiteEventLogger mints Guid.NewGuid("N") per event and inserts it.
Task 5b - three consumers assumed a monotonic integer id. All three move to
timestamp ordering; leaving any one behind would be a live bug:
- EventLogQueryService: "id > $afterId" would return an ARBITRARY subset of a
GUID-keyed table and SILENTLY DROP ROWS from page-through. Now a composite
(timestamp, id) keyset cursor with an opaque string token; timestamps are not
unique, so id is the tie-break that guarantees exactly-once paging.
- EventLogPurgeService: "ORDER BY id ASC LIMIT 1000" would delete a RANDOM batch
instead of the oldest. Now orders by timestamp.
- EventLogEntry.Id and both ContinuationTokens: long -> string.
WIRE COMPATIBILITY. Those DTOs cross the site<->central Akka boundary
(SiteCommunicationActor -> CommunicationService -> ManagementActor / CentralUI).
No rolling-upgrade shim is needed because both sides ship in the same deployable
and the rig redeploys as a unit. Checked: no Akka serializer binding pins these
types by name. A stale numeric token degrades to "start from the beginning"
(a visible repeat) rather than throwing or losing rows.
Tests: the two that encoded the old semantics were rewritten to guard the new
invariant rather than deleted - uniqueness instead of monotonicity, and
oldest-purged-first keyed on timestamp. That second test also exposed a latent
weakness: the bulk seed stamped every row with the same UtcNow, so "oldest" was
never actually well-defined; rows now get distinct increasing timestamps, kept
inside the retention window so the retention purge does not eat them first.
Verified:
dotnet build ZB.MOM.WW.ScadaBridge.slnx -> 0 Error(s), 0 Warning(s)
SiteEventLogging.Tests -> 70 passed (12 new schema tests, red-first)
CentralUI.Tests -> 925 passed
Commons.Tests -> 684 passed
SiteRuntime.Tests -> 529 passed, 1 pre-existing flaky failure
(InstanceActorChildAttributeRaceTests - passes 3/3 in
isolation with and without this change)
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Gitea #18: the process-wide ScriptExecutionScheduler was reached via a static
singleton (Shared) directly inside ScriptActor, ScriptExecutionActor, AlarmActor,
AlarmExecutionActor, and ScriptSchedulerStatsReporter, so it could not be
substituted — a shared mutable global for anything running more than one logical
site in a process, most visibly the test assembly. Add an optional scheduler
injection seam to all five: the Host injects nothing and gets the shared pool
(byte-for-byte unchanged), while a test (or a future multi-site host) hands an
actor its own instance. Spawning actors thread their scheduler down to the
execution children they create. Shared() now also recreates a disposed cached
instance rather than returning it (new IsDisposed guard), so a disposed scheduler
can never silently poison every later script/alarm execution.
Gitea #16: ScriptActorTests now runs on its OWN scheduler (disposed at class
teardown), so the faulted-task test can no longer strand a worker on the
process-wide pool and starve unrelated test classes (one prior run failed 22
tests). EvalGate's wait is bounded to 30 s (was unbounded — the actual leak
mechanism) and reset per test; the teardown releases one permit per worker
instead of Release(Entries), which under-released in exactly the failure case.
Full SiteRuntime suite: 6/6 runs green (524/524); was 3/6 failing on clean main.
Fixes: #18Fixes: #16
SiteScriptCompileCache.Hits is a process-global counter; other test classes in
the assembly compile scripts concurrently (not in this serialized collection),
so an exact post-Clear count is non-deterministic under a full-assembly parallel
run. Assert.Same(r1,r2) remains the definitive proof of one shared Roslyn
compile. Integration-surfaced; belongs with plan R2-03 T5/T6.
VERIFY-THEN-FIX: the plan-authoring discovery is CONFIRMED. No code anywhere in
src/ registered IOperationTrackingStore in DI, yet AkkaHostedService and
AuditLog SCE both comment that AddSiteRuntime provides it. Every consumer
resolved it via GetService (null-tolerant) and silently ran degraded:
cached-drain scheduler never armed, PullSiteCalls reconciliation seam never
wired, Tracking.Status degraded to audit-only. This is a FUNCTIONAL FIX riding
the hygiene plan: AddSiteRuntime now registers the store (site-only), so
site-local cached-call tracking runs in its intended mode. OperationTrackingOptions
is now bound + eagerly validated in the Host site-options block.
Adds SiteStreamManager.SubscribeSiteAlarms(IActorRef) — same broadcast-hub
wiring as Subscribe(...) but with no per-instance filter and forwarding only
AlarmStateChanged events (AttributeValueChanged dropped). Returns a subscription
id torn down via the existing Unsubscribe, symmetric with the per-instance
subscribe. Backs the Task 2 SubscribeSite gRPC server handler.
Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
Site side: additive proto after_id field, ReadChangedSinceAsync gains an optional
afterId cursor and deterministic (UpdatedAtUtc, TrackedOperationId) ordering — a
batch fully inside one UpdatedAtUtc instant no longer re-reads the same page
forever. Absent/empty afterId preserves the exact legacy inclusive >= contract, so
an older central is unaffected. Regenerated the checked-in gRPC C#.
Success now requires BOTH persistence commit AND an InstanceActorInitialized
readiness signal from the actor's PreStart — persistence can commit before the
actor's async init has run or failed, so persistence alone must not report
Success. An actor that dies during init never signals readiness; the Terminated
fallback fails that deployment and rolls back the optimistic state. The
persisted-row rollback is deferred until the store commits so it cannot race the
optimistic write.
Deviation from plan Task 15: the plan's swallow-only guard did not handle the
persistence-first ordering (empirically the store commits before the Terminated
signal, so Success was reported for a dead actor). Added the readiness handshake
to make the join deterministic.
Validation runs on the actor thread (not off-thread as the plan sketched): piping
the verdict back to self reorders concurrent deploys relative to each other and to
delete/disable, breaking the redeploy-supersede FIFO ordering (SR020/SR029). A deploy
is an infrequent admin command, so a brief synchronous Roslyn compile is acceptable.
Joseph Doherty