Commit Graph

3348 Commits

Author SHA1 Message Date
Joseph Doherty 445bb855e2 fix: release the alarm drain latch on escape; cancel before disposing the page poll timers 2026-08-16 05:40:27 -04:00
Joseph Doherty b9e9df4081 docs(code): honest rationales — Akka restart semantics, per-key shards, EF literal inlining; retire or annotate orphaned members 2026-08-16 05:33:33 -04:00
Joseph Doherty dbc9223206 docs(code): correct comments stale after the perf remainder 2026-08-16 05:09:00 -04:00
Joseph Doherty e75bb38734 docs: perf roadmap remainder — outcomes + follow-up register 2026-08-16 05:08:53 -04:00
Joseph Doherty 451273c860 test(historian): pin the failed-batch-append branch; correct its loss-accounting docs
The batched ingest's append-failure catch had zero coverage, and two doc
statements overstated the loss: the Warning said the burst's values were
"dropped", and RecorderStatus.OutboxAppendFailures said "the value was
dropped, not recorded". The outbox contract is prefix-never-orphan, so a
partial-prefix failure leaves entries durably queued that drain normally.

- New test pins the branch: one 3-item burst whose group commit commits a
  1-entry prefix then throws => OutboxAppendFailures == 1 (per burst, not
  per value), TotalRecorded == 0, actor alive, and the prefix still reaches
  the writer via the post-failure DrainTick (drainInterval 30s, so the
  steady cadence cannot account for it). Falsified twice: counting per item
  and dropping the drain nudge each turn it red.
- Second test pins the OperationCanceledException filter from its
  observable side - a cancel that is NOT shutdown is metered, not swallowed.
  Removing the 'when (_lifetime.IsCancellationRequested)' turns it red. The
  shutdown side has no cheap seam (the counter dies with the actor).
- Docs corrected: OutboxAppendFailures counts FAILED BATCH APPENDS, one per
  burst whatever its size (a granularity change from the per-value era), and
  is not a count of values lost; TotalRecorded under-counts when a commit
  fails after a partial prefix. Log message says "failed to append" and
  names the prefix. No functional change beyond the message string.

Also folds in the DrainBatchSize default alignment behind 0298a0b9: the
recorder's two ctor defaults and its non-positive fallback still said 64
while ContinuousHistorizationOptions.DrainBatchSize now defaults to 1024.
No test pinned the old 64.
2026-08-16 04:50:30 -04:00
Joseph Doherty 0298a0b944 chore(historian): batch-era defaults + appsettings/doc alignment 2026-08-16 04:48:55 -04:00
Joseph Doherty 5b92eb0ff6 perf(runtime): direct-mode artifact read piped off the host mailbox 2026-08-16 04:48:02 -04:00
Joseph Doherty ca87329a83 fix(adminui): dedup + chunk the rename script-scan OR-chain 2026-08-16 04:36:03 -04:00
Joseph Doherty 485f4ea1e8 test(polling): tolerate one scheduler blip in the fast-reader overrun control 2026-08-16 04:35:01 -04:00
Joseph Doherty 384568b47a perf(historian): batch ingest — one group commit per value burst 2026-08-16 04:33:35 -04:00
Joseph Doherty aad7583b79 test(acl): pin TranslateBrowsePath survivor order through the compaction 2026-08-16 04:29:27 -04:00
Joseph Doherty b6275f439c perf(runtime): per-driver index for connectivity fans; comms-loss is one batch 2026-08-16 04:27:20 -04:00
Joseph Doherty 9cf400f1e8 perf(drivers): fixed-rate poll pacing with overrun counter; non-blocking unsubscribe 2026-08-16 04:26:20 -04:00
Joseph Doherty 684e59d871 perf(adminui): drift chip recomposes only when the config-edit watermark moves 2026-08-16 04:25:56 -04:00
Joseph Doherty d5f573b019 fix(abcip): single-flight cold-connect per device — backoff survives the parallel read window 2026-08-16 04:24:59 -04:00
Joseph Doherty fe8d132c64 perf(adminui): rename script scan filters server-side, stops shipping every script body 2026-08-16 04:24:10 -04:00
Joseph Doherty a76b4e0740 perf(historian): bounded-parallel per-tag drain (DrainConcurrency, default 8) 2026-08-16 04:21:18 -04:00
Joseph Doherty d8a17f2e34 perf(acl): authorizer hot path — cache lookup before scope build, lazy deny reasons 2026-08-16 04:19:47 -04:00
Joseph Doherty 7320586512 fix(historian): incremental batch-append bookkeeping; genuine group-commit durability pin 2026-08-16 04:18:38 -04:00
Joseph Doherty fbe8aaad9d perf(acl): dedup subscription revalidation by (session, node) 2026-08-16 04:18:03 -04:00
Joseph Doherty 50035b36bb test(mqtt): pin status-code half of republish suppression; note Sparkplug bypass 2026-08-16 04:16:35 -04:00
Joseph Doherty 39f67a0fce perf(s7): span-grouped reads — one PDU per contiguous block instead of one per tag 2026-08-16 04:15:37 -04:00
Joseph Doherty 63081d52d8 fix(adminui): fleet page disposes its poll timer asynchronously like hosts 2026-08-16 04:15:05 -04:00
Joseph Doherty 409a58b624 fix(modbus): lifecycle gate + non-vacuous reprobe regression test 2026-08-16 04:07:27 -04:00
Joseph Doherty 140bf0c131 perf(adminui): pooled DbContext factory; role-grants reads per-call, not per-circuit 2026-08-16 04:05:34 -04:00
Joseph Doherty 34f836e03c perf(mtconnect): sample chunks parse from bytes, no per-chunk string copies 2026-08-16 04:05:15 -04:00
Joseph Doherty 3795899d10 perf(adminui): 5s snapshot cache in front of the script tag catalog 2026-08-16 04:05:07 -04:00
Joseph Doherty ba977dbe0d perf(adminui): raw tree renders tag rows from a projection without TagConfig 2026-08-16 04:04:32 -04:00
Joseph Doherty fa36d64e8f perf(adminui): one guard round-trip per bulk reference add 2026-08-16 04:03:49 -04:00
Joseph Doherty 541a223b96 perf(acl): O(n) browse/translate filtering compaction 2026-08-16 04:03:32 -04:00
Joseph Doherty 4b6a93bb78 fix(adminui): fleet/hosts poll ticks no longer overlap; cancel on dispose 2026-08-16 04:03:31 -04:00
Joseph Doherty d6a4fbd648 perf(abcip): bounded-parallel tag reads (MaxConcurrentReads, default 8) 2026-08-16 04:02:40 -04:00
Joseph Doherty 5df793e79a perf(alarms): dirty-set coalescing — one evaluation per alarm per drain, latest values win 2026-08-16 04:02:20 -04:00
Joseph Doherty 2c708f5560 perf(mqtt): suppress unchanged-value republish fan-out 2026-08-15 21:42:00 -04:00
Joseph Doherty 8d91833fed feat(s7): pure narrow-scalar decode from byte spans 2026-08-15 21:41:46 -04:00
Joseph Doherty 6ff64ca067 perf(historian): group-commit batch append on the historization outbox 2026-08-15 21:40:04 -04:00
Joseph Doherty debd44c213 perf(acl): per-session decision-cache shards — a crawl no longer evicts the fleet 2026-08-15 21:38:48 -04:00
Joseph Doherty dbc9a132e2 perf(adminui): /deployments lists a projection, not 50 artifact blobs 2026-08-15 21:38:11 -04:00
Joseph Doherty c3970a2e5d fix(modbus): dispose the previous transport and probe loops on re-initialize 2026-08-15 21:37:28 -04:00
Joseph Doherty dc459633e0 docs(plan): perf roadmap remainder — historian batching, ACL cache, driver round-trips, alarm coalescing, AdminUI hot spots, residuals 2026-08-15 21:33:18 -04:00
Joseph Doherty a8c93a0080 docs: record surgical-deploys docker-dev live gate (PASSED) + correct rename-vs-identity rebuild wording
v2-ci / build (push) Successful in 5m5s
v2-ci / unit-tests (push) Failing after 17m48s
2026-08-15 20:01:37 -04:00
Joseph Doherty 169ee6a465 Merge branch 'perf/surgical-deploys' — surgical deploy applies (classifier whitelist + delta materialisation), parse-once artifact, gzip artifact blob, stash + wedge watchdog
v2-ci / build (push) Successful in 5m5s
v2-ci / unit-tests (push) Failing after 17m19s
2026-08-15 19:42:01 -04:00
Joseph Doherty 5e336114c4 Merge branch 'perf/value-pipeline-batching' — end-to-end value-path batching (IBatchSubscribable, batched actor messages, single-lock chunked WriteValues, RealmKey re-key, dispatcher throughput 100) 2026-08-15 19:42:01 -04:00
Joseph Doherty 3122728593 test(runtime): serialize meter-asserting publish-actor suites (cross-collection counter pollution) 2026-08-15 18:16:59 -04:00
Joseph Doherty 0b537b5dec docs: record surgical-deploy semantics + artifact compression ops notes (plan 2026-08-15)
CLAUDE.md — a new "How a deploy reaches that address space" block under the v3
Batch 4 address-space section. The blanket ChangedRawTags/ChangedUnsReferenceVariables
=> Rebuild rule is gone, so the doc now names both halves explicitly: what still
rebuilds (alarm-bearing raw-tag deltas, raw-tag identity changes, raw container
renames/removals, changed equipment/alarms) and what is now surgical (non-alarm
attribute deltas, every UnsReferenceDelta). Also records the three semantics that
are easy to break by accident: mirror repair rides UnsReferenceVariable's inherited
backing fields (omitting a newly-inherited field silently reopens the staleness),
a degraded non-rebuild apply deliberately does not advance _lastApplied, and the
raw-realm announce gap is no longer neutral now that surgical adds keep
subscriptions alive.

CLAUDE.md ConfigSource — the RevisionHash == SHA-256(artifact bytes) paragraph is
still true and stays; it gains the clause that the blob is gzip-compressed since
2026-08-15 and the hash covers the COMPRESSED bytes, which is precisely why the
fetcher, cache and serve path were untouched.

docs/Uns.md + docs/Raw.md — one section each on what a deploy now tears down, from
the authoring surface's point of view, including the mirror repair semantics
(identity-preserving backing edits update in place and keep their MonitoredItems;
re-points and effective-name edits remove+recreate that one node).

docs/Configuration.md — "Artifact compression (2026-08-15)" under ConfigSource/
ConfigServe: gzip inside the hashed bytes, magic-sniffed on read, pre-upgrade
uncompressed rows still parse, plus the two operational facts an operator needs.
The first deploy after upgrade re-hashes every revision (same config, new bytes,
new RevisionHash) so it is a one-time full re-dispatch rather than a no-op; and
rollout is ordered — nodes must run artifact-sniffing code BEFORE central starts
emitting compressed blobs, since a pre-upgrade node handed a gzip artifact fails
the apply and keeps last-known-good until it is upgraded.

Plan doc — a "Follow-ups (recorded, not implemented)" section covering the seven
items surfaced during implementation and review: the raw-realm announce gap, the
RemoveEquipmentSubtree dangling-inverse-edge exposure, the missing end-to-end test
for a degraded remove-bearing apply's ratcheted rebuild, the scripted-alarm
thin-plan-ids-against-rich-rows blindness, the absent monotonic dispatch sequence,
LocalDb store ordering under concurrent StoreAsync, and threading the in-hand blob
through RestoreApplied.

tasks.json — all nine tasks completed.

Docs only; no source changes.
2026-08-15 18:02:21 -04:00
Joseph Doherty d787502d54 fix(opcuaserver): degraded applies re-diff instead of losing nodes; in-place UNS mirror updates; sever inverse edges
C1 — a non-rebuild apply reporting failedNodes > 0 no longer advances _lastApplied.
Pre-delta-materialisation the full passes re-ensured every node on every apply, which
silently repaired any prior swallowed sink fault. The delta path removed that accidental
self-healing: a failed node sat inside _lastApplied, never appeared in a diff set again,
and EnsureVariable early-returns on the ones that landed — so it was lost until an
unrelated change forced a rebuild. The advance also moved to AFTER the materialise
passes, so a throwing pass leaves _lastApplied at the previous composition too. A
rebuild still advances even with failures (the ratchet is its resync path).

I2 — a UnsReferenceDelta whose identity is unchanged (same NodeId, EquipmentId and
BackingRawPath) is now applied in place via UpdateTagAttributes(Uns) instead of
remove+recreate, so the mirror's MonitoredItems survive an operator historizing or
reshaping the raw tag behind them. Remove+recreate stays for identity-changing deltas
(re-point, effective-name edit).

I3 — RemoveVariableNode severs the inverse half of every outgoing Organizes edge, the
variable-node counterpart of UnwireAlarmNotifiers. Rebuilds used to sweep these up;
surgical applies made rebuilds rare, so they accumulated on the raw node for the life of
the process.

I5 — added UNS reference variables are announced (their equipment folder is the parent,
already the Uns realm). Raw adds still are not; the gap comment now says so honestly
rather than calling it neutral and pre-existing.

I4 + M1 — corrected two comments that no longer described the code: the "re-point →
rebuild" claim in ApplyPureRemove, and the "N lock acquisitions" rationale for the delta
passes (Ensure* early-return on a dictionary probe before the lock).
2026-08-15 17:53:26 -04:00
Joseph Doherty 0dbbb5ac6b fix(runtime): stash dedup + applying-wedge watchdog; cache-failure diagnostics 2026-08-15 17:43:15 -04:00
Joseph Doherty bfbdce32e4 perf(opcuaserver): non-rebuild applies materialise the delta, not the composition 2026-08-15 17:26:53 -04:00
Joseph Doherty 14c31941c7 fix(runtime): parallel-safe decompression limit; tighten ParsedDeploymentArtifact surface 2026-08-15 17:16:09 -04:00
Joseph Doherty 5f2e60278c perf(runtime): apply-path cache write off-mailbox; stash deferred dispatches 2026-08-15 17:12:01 -04:00