Merge grpc-deltep-probe: DelTep multiplexed-channel probe (disproven)

gRPC's single shared channel does NOT lift the WCF per-connection working-set
wall for DeleteTagExtendedProperties — probed live 2026-06-22, primes succeed
but the delete is still rejected (native code=1). Stays server-blocked on both
transports, unshipped; pinned by a gated negative test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6mcaT2PjRFKcogzp9UkfC

README.md

@@ -91,7 +91,7 @@ request rides the RPC but the server faults on an unmet precondition) ·
 | `ReadEventsAsync` | ✅ | ⚠️ | tooled + routed over gRPC: the full CM_EVENT registration replay (`UpdateClientStatus`→`RegisterTags`→`EnsureTags` + discovery probes) runs and `StartEventQuery` succeeds, but `GetNextEventQueryResultBuffer` **long-polls** on no data (it blocks to the deadline rather than returning the synchronous 5-byte code-85 terminal the WCF op gives). The read is **hard-bounded** (≤30s) and throws `ProtocolEvidenceMissingException` on the no-row path rather than assert a false empty. Row-level retrieval is **not yet live-verified** — the dev box holds no events; pending a capture against an event-bearing 2023 R2 server. Use WCF for event reads |
 | `SendEventAsync` | ✅ | 🔌 | rides `AddStreamValues` family; no distinct event-send RPC, framing uncaptured over gRPC |
 | `EnsureTagAsync` / `DeleteTagAsync` / `RenameTagsAsync` | ✅ | ✅ | live-verified 2026-06-22 over gRPC (`HistoryService.EnsureTags` / `DeleteTags` / `StartJob`, write-enabled 0x401 session, WCF serializers reused) via a self-cleaning sandbox-tag lifecycle. Rename is an async StartJob — transiently rejectable right after create, so callers should retry |
-| `AddTagExtendedPropertiesAsync` | ✅ | ✅ | live-verified 2026-06-22 over gRPC (`HistoryService.AddTagExtendedProperties`, write-enabled session); a written prop now round-trips through `GetTagExtendedPropertiesAsync` (the multi-property parser fix above). gRPC also exposes `DeleteTagExtendedProperties` (WCF delete was server-blocked) |
+| `AddTagExtendedPropertiesAsync` | ✅ | ✅ | live-verified 2026-06-22 over gRPC (`HistoryService.AddTagExtendedProperties`, write-enabled session); a written prop now round-trips through `GetTagExtendedPropertiesAsync` (the multi-property parser fix above). `DeleteTagExtendedProperties` stays unshipped: probed over gRPC 2026-06-22 (prime `GetTgByNm`+`GetTepByNm` then `DelTep`, all on the one shared channel) — the server still rejects the delete (native code=1) and the property survives, so gRPC's multiplexed channel does **not** lift the WCF per-connection working-set wall |
 | `GetConnectionStatusAsync` | ✅ | ✅ | live-verified 2026-06-22 over gRPC — measured from the handshake (`OpenConnection` yields a storage-session GUID ⇒ connected). No dedicated RPC on either transport; store-forward connectivity stays false (D2-gated) |
 | `ReadBlocksAsync` | ❌ | ❌ | `StartBlockRetrievalQuery` never captured on either transport — throws `ProtocolEvidenceMissingException` |
 

docs/plans/grpc-tooling-completion.md

@@ -140,8 +140,19 @@ _Original notes (still the reference for the registration replay):_
 ### Out of scope
 - `ReadBlocks` (`StartBlockRetrievalQuery`) — never captured on either transport; leave
   throwing `ProtocolEvidenceMissingException`.
-- `DeleteTagExtendedProperties` — server-blocked on WCF (per-connection working set);
-  gRPC's single multiplexed channel *might* fix it — opportunistic probe only.
+- `DeleteTagExtendedProperties` — ❌ **PROBED 2026-06-22, multiplexed-channel hypothesis DISPROVEN.**
+  The WCF block (server resolves the property from a per-connection working set the SDK's separate
+  per-service channels can't populate) is NOT lifted by gRPC. The probe
+  (`HistorianGrpcTagWriteOrchestrator.ProbeDeleteTagExtendedPropertiesAsync`) runs the native
+  `GetTgByNm` → `GetTepByNm` → `DelTep` sequence over ONE write-enabled (0x401) session on gRPC's
+  single shared channel. Live against the 2023 R2 server (History iface 12): both primes succeed on the
+  shared channel (`TgPrimeBytes=98`, `TepPrimePages=1`) yet `DelTep` is still rejected with native
+  **code=1** (the 5-byte error buffer's byte0=132 is the universal `0x84` marker, not a code) and the
+  property survives. Conclusion: the working set the server consults is populated by something the SDK
+  can't reproduce even over one connection — most likely the native client's in-process registration
+  object, not the wire session. Stays server-blocked on BOTH transports; not shipped publicly. Pinned
+  by the gated negative test `DeleteTagExtendedProperties_OverGrpc_ProbeMultiplexedChannel` (flips if a
+  future server/registration lifts the wall).
 
 ## Live verification setup (every live run)
 

src/AVEVA.Historian.Client/Grpc/HistorianGrpcTagWriteOrchestrator.cs

@@ -2,6 +2,7 @@ using Google.Protobuf;
 using AVEVA.Historian.Client.Models;
 using AVEVA.Historian.Client.Wcf;
 using GrpcHistory = ArchestrA.Grpc.Contract.History;
+using GrpcRetrieval = ArchestrA.Grpc.Contract.Retrieval;
 
 namespace AVEVA.Historian.Client.Grpc;
 
@@ -146,6 +147,114 @@ internal sealed class HistorianGrpcTagWriteOrchestrator
         return response.Status?.BSuccess ?? false;
     }
 
+    /// <summary>Outcome of the <see cref="ProbeDeleteTagExtendedPropertiesAsync"/> single-channel delete probe.</summary>
+    /// <param name="Accepted">True if the server's <c>DelTep</c> returned success.</param>
+    /// <param name="ErrorDescription">Decoded native error (byte0 0x84 + LE code + facility/file/message) when rejected.</param>
+    /// <param name="TagInfoPrimeBytes">Bytes returned by the GetTgByNm prime (tag-identity working-set load).</param>
+    /// <param name="ExtPropPrimePages">GetTepByNm prime pages that returned success (extended-property working-set load).</param>
+    internal readonly record struct DeleteTagExtendedPropertiesProbeResult(
+        bool Accepted, string? ErrorDescription, int TagInfoPrimeBytes, int ExtPropPrimePages);
+
+    /// <summary>
+    /// <b>Reverse-engineering probe (not a public op).</b> Tests whether <c>DelTep</c>
+    /// (DeleteTagExtendedProperties) — server-blocked on the 2020 WCF transport — succeeds over gRPC.
+    /// The WCF failure is structural: the server's <c>CHistStorage::DeleteTagExtendedProperties</c>
+    /// resolves each property from a <i>per-connection working set</i> the native client populates by
+    /// multiplexing <c>GetTgByNm</c> + <c>GetTepByNm</c> + <c>DelTep</c> over ONE physical connection.
+    /// The WCF SDK uses a separate channel per service, so the prime and the delete never share a
+    /// connection and the working set is empty at delete time (SErrorException). Over gRPC every service
+    /// client is built on the SAME <see cref="HistorianGrpcConnection.Channel"/>, so this probe runs the
+    /// identical native sequence — GetTgByNm prime, GetTepByNm prime, then DelTep — on one write-enabled
+    /// (0x401) session/channel, to see whether the multiplexed channel satisfies the working-set check.
+    /// Returns the decoded outcome rather than throwing so the caller can record a positive or negative
+    /// result. See docs/reverse-engineering/wcf-add-tag-extended-properties.md §Delete.
+    /// </summary>
+    internal Task<DeleteTagExtendedPropertiesProbeResult> ProbeDeleteTagExtendedPropertiesAsync(
+        string tagName, IReadOnlyList<string> propertyNames, CancellationToken cancellationToken)
+    {
+        ArgumentException.ThrowIfNullOrWhiteSpace(tagName);
+        ArgumentNullException.ThrowIfNull(propertyNames);
+        if (propertyNames.Count == 0)
+        {
+            throw new ArgumentException("At least one property name is required.", nameof(propertyNames));
+        }
+        return Task.Run(() => ProbeDeleteTagExtendedProperties(tagName, propertyNames, cancellationToken), cancellationToken);
+    }
+
+    private DeleteTagExtendedPropertiesProbeResult ProbeDeleteTagExtendedProperties(
+        string tagName, IReadOnlyList<string> propertyNames, CancellationToken cancellationToken)
+    {
+        using HistorianGrpcConnection connection = HistorianGrpcChannelFactory.Create(_options);
+        HistorianGrpcHandshake.Session session = HistorianGrpcHandshake.OpenSession(connection, _options, cancellationToken, WriteEnabledConnectionMode);
+
+        var retrievalClient = new GrpcRetrieval.RetrievalService.RetrievalServiceClient(connection.Channel);
+        var historyClient = new GrpcHistory.HistoryService.HistoryServiceClient(connection.Channel);
+        DateTime Deadline() => DateTime.UtcNow.Add(_options.RequestTimeout);
+
+        // Prime 1 — GetTgByNm: load the tag identity into the storage session's working set (same channel).
+        int tagInfoPrimeBytes = 0;
+        try
+        {
+            GrpcRetrieval.GetTagInfosFromNameResponse tg = retrievalClient.GetTagInfosFromName(
+                new GrpcRetrieval.GetTagInfosFromNameRequest
+                {
+                    StrHandle = session.StringHandle,
+                    BtTagNames = ByteString.CopyFrom(HistorianGrpcTagClient.BuildTagNamesBuffer([tagName])),
+                    UiSequence = 0
+                },
+                connection.Metadata, Deadline(), cancellationToken);
+            tagInfoPrimeBytes = tg.BtTagInfos?.Length ?? 0;
+        }
+        catch
+        {
+            // Best-effort prime; the delete still runs so the error buffer is captured.
+        }
+
+        // Prime 2 — GetTepByNm: load the tag's extended properties into the working set (same channel),
+        // exactly as the native register->read->delete sequence does on its single connection.
+        int extPropPrimePages = 0;
+        byte[] tepRequest = HistorianTagExtendedPropertyProtocol.SerializeRequest(tagName);
+        uint sequence = 0;
+        for (int page = 0; page < 64; page++)
+        {
+            cancellationToken.ThrowIfCancellationRequested();
+            GrpcRetrieval.GetTagExtendedPropertiesFromNameResponse tep = retrievalClient.GetTagExtendedPropertiesFromName(
+                new GrpcRetrieval.GetTagExtendedPropertiesFromNameRequest
+                {
+                    StrHandle = session.StringHandle,
+                    BtTagNames = ByteString.CopyFrom(tepRequest),
+                    UiSequence = sequence
+                },
+                connection.Metadata, Deadline(), cancellationToken);
+            if (!(tep.Status?.BSuccess ?? false))
+            {
+                break;
+            }
+            extPropPrimePages++;
+            if (HistorianTagExtendedPropertyProtocol.ParseResponse(tep.BtTeps?.ToByteArray() ?? []).Count == 0)
+            {
+                break;
+            }
+            sequence = tep.UiSequence;
+        }
+
+        // DelTep on the SAME channel/session, while the priming reads are part of the same working set.
+        byte[] inBuff = HistorianTagExtendedPropertyProtocol.SerializeDeleteRequest(tagName, propertyNames);
+        GrpcHistory.DeleteTagExtendedPropertiesResponse delete = historyClient.DeleteTagExtendedProperties(
+            new GrpcHistory.DeleteTagExtendedPropertiesRequest
+            {
+                StrHandle = session.StringHandle,
+                BtInput = ByteString.CopyFrom(inBuff)
+            },
+            connection.Metadata, Deadline(), cancellationToken);
+
+        bool accepted = delete.Status?.BSuccess ?? false;
+        string? errorDescription = accepted
+            ? null
+            : HistorianEventRegistrationProtocol.DescribeNativeError(delete.Status?.BtError?.ToByteArray() ?? []);
+        return new DeleteTagExtendedPropertiesProbeResult(accepted, errorDescription, tagInfoPrimeBytes, extPropPrimePages);
+    }
+
     public Task<HistorianTagRenameResult> RenameTagsAsync(
         IReadOnlyList<(string OldName, string NewName)> pairs, CancellationToken cancellationToken)
     {

src/AVEVA.Historian.Client/HistorianClient.cs

@@ -256,9 +256,14 @@ public sealed class HistorianClient : IAsyncDisposable
     // golden-verified against a server-accepted buffer, but the SDK cannot yet make the 2020 server
     // accept the delete: the server's CHistStorage::DeleteTagExtendedProperties consults a
     // per-connection working set that the native client populates by multiplexing GetTepByNm and
-    // DelTep over ONE connection, which the SDK's per-service WCF channels don't reproduce. See the
-    // documented-but-blocked path in HistorianWcfTagWriteOrchestrator and
-    // docs/reverse-engineering/wcf-add-tag-extended-properties.md §Delete.
+    // DelTep over ONE connection, which the SDK's per-service WCF channels don't reproduce. The gRPC
+    // transport — where every service client shares ONE channel — was probed 2026-06-22 to test that
+    // multiplexing hypothesis (GetTgByNm + GetTepByNm prime then DelTep on one write-enabled session,
+    // HistorianGrpcTagWriteOrchestrator.ProbeDeleteTagExtendedPropertiesAsync): both primes succeed on
+    // the shared channel yet the server STILL rejects the delete (native code=1), so gRPC does not lift
+    // the wall either. The working set is evidently populated by the native client's in-process
+    // registration state, not the wire session. See the documented-but-blocked path in
+    // HistorianWcfTagWriteOrchestrator and docs/reverse-engineering/wcf-add-tag-extended-properties.md §Delete.
 
     /// <summary>
     /// Executes a SQL command against the Historian over the WCF <c>ExeC</c>/<c>GetR</c> ops and

tests/AVEVA.Historian.Client.Tests/HistorianGrpcIntegrationTests.cs

@@ -531,6 +531,85 @@ public sealed class HistorianGrpcIntegrationTests
         Assert.False(status.ConnectedToStoreForward);
     }
 
+    [Fact]
+    public async Task DeleteTagExtendedProperties_OverGrpc_ProbeMultiplexedChannel()
+    {
+        string? host = Environment.GetEnvironmentVariable("HISTORIAN_GRPC_HOST");
+        // DESTRUCTIVE: reuses the write sandbox-tag gate so it never mutates a server by accident.
+        string? sandbox = Environment.GetEnvironmentVariable("HISTORIAN_GRPC_WRITE_SANDBOX_TAG");
+        if (string.IsNullOrWhiteSpace(host) || string.IsNullOrWhiteSpace(sandbox)
+            || string.IsNullOrEmpty(Environment.GetEnvironmentVariable("HISTORIAN_USER")))
+        {
+            return;
+        }
+
+        // Out-of-scope opportunistic probe (grpc-tooling-completion.md). DelTep is server-blocked on the
+        // 2020 WCF transport because the server resolves the property from a per-CONNECTION working set
+        // that WCF's separate per-service channels can't populate (prime + delete land on different
+        // connections). gRPC builds every service client on ONE shared channel, so this probe runs the
+        // native GetTgByNm -> GetTepByNm -> DelTep sequence over a single write-enabled session to see
+        // whether the multiplexed channel satisfies the working-set check.
+        //
+        // RESULT — live 2026-06-22 (2023 R2 server, History iface 12): the multiplexed channel does NOT
+        // lift the wall. BOTH primes succeed on the shared channel (GetTgByNm returns the tag identity,
+        // GetTepByNm returns the property page), yet DelTep is still rejected with native error code=1
+        // (the 5-byte buffer's byte0=132 is the universal 0x84 error marker, not a code) and the property
+        // survives. So DelTep stays server-blocked on BOTH transports — the working set the server
+        // consults is populated by something the SDK can't reproduce even over one connection (likely the
+        // native client's in-process registration object, not the wire session). This test PINS that
+        // negative: it asserts the probe is rejected and the property survives. If a future server or
+        // registration change lifts the wall, the inverted assertion flips and we notice.
+        HistorianClientOptions options = BuildOptions(host);
+        HistorianClient client = new(options);
+        const string propName = "GrpcDelProbe";
+
+        try
+        {
+            try { await client.DeleteTagAsync(sandbox!, CancellationToken.None); } catch { /* clean slate */ }
+
+            Assert.True(await client.EnsureTagAsync(
+                new HistorianTagDefinition { TagName = sandbox!, DataType = HistorianDataType.Float, EngineeringUnit = "u", MaxEU = 100 },
+                CancellationToken.None), "EnsureTags should create the sandbox tag.");
+            Assert.True(await client.AddTagExtendedPropertyAsync(sandbox!, propName, "todelete", CancellationToken.None),
+                "AddTagExtendedProperties should seed the property to delete.");
+
+            IReadOnlyList<HistorianTagExtendedProperty> before = await client.GetTagExtendedPropertiesAsync(sandbox!, CancellationToken.None);
+            Assert.Contains(before, p => string.Equals(p.Name, propName, StringComparison.OrdinalIgnoreCase));
+
+            HistorianGrpcTagWriteOrchestrator orchestrator = new(options);
+            HistorianGrpcTagWriteOrchestrator.DeleteTagExtendedPropertiesProbeResult probe =
+                await orchestrator.ProbeDeleteTagExtendedPropertiesAsync(sandbox!, [propName], CancellationToken.None);
+
+            // The primes must succeed (proving the single channel DID load the working set) — otherwise the
+            // negative below would be inconclusive (a prime failure, not a delete-path block).
+            Assert.True(probe.TagInfoPrimeBytes > 0, "GetTgByNm prime should return the tag identity on the shared channel.");
+            Assert.True(probe.ExtPropPrimePages > 0, "GetTepByNm prime should return the property page on the shared channel.");
+
+            IReadOnlyList<HistorianTagExtendedProperty> after = await client.GetTagExtendedPropertiesAsync(sandbox!, CancellationToken.None);
+            bool stillPresent = after.Any(p => string.Equals(p.Name, propName, StringComparison.OrdinalIgnoreCase));
+
+            // Pinned negative: the multiplexed channel does not lift the working-set wall.
+            Assert.False(probe.Accepted,
+                $"Unexpected: DelTep over gRPC was ACCEPTED — the multiplexed channel may now lift the wall. " +
+                $"Promote DeleteTagExtendedProperties to a public op. Probe={DescribeProbe(probe, stillPresent)}");
+            Assert.True(stillPresent,
+                $"Unexpected: property was removed despite DelTep reporting failure. Probe={DescribeProbe(probe, stillPresent)}");
+        }
+        finally
+        {
+            for (int i = 0; i < 5; i++)
+            {
+                try { await client.DeleteTagAsync(sandbox!, CancellationToken.None); } catch { /* best-effort */ }
+                await Task.Delay(TimeSpan.FromSeconds(1));
+            }
+        }
+    }
+
+    private static string DescribeProbe(
+        HistorianGrpcTagWriteOrchestrator.DeleteTagExtendedPropertiesProbeResult probe, bool stillPresent)
+        => $"Accepted={probe.Accepted} StillPresent={stillPresent} TgPrimeBytes={probe.TagInfoPrimeBytes} " +
+           $"TepPrimePages={probe.ExtPropPrimePages} Err={probe.ErrorDescription}";
+
     private static HistorianClientOptions BuildOptions(string host)
     {
         string? user = Environment.GetEnvironmentVariable("HISTORIAN_USER");